| *** tosky has quit IRC | 00:26 | |
| *** yamamoto has joined #openstack-meeting-3 | 00:32 | |
| *** macz_ has quit IRC | 00:39 | |
| *** yamamoto has quit IRC | 00:39 | |
| *** yamamoto has joined #openstack-meeting-3 | 00:48 | |
| *** yamamoto has quit IRC | 00:48 | |
| *** yamamoto has joined #openstack-meeting-3 | 00:49 | |
| *** yamamoto has quit IRC | 00:54 | |
| *** yamamoto has joined #openstack-meeting-3 | 01:37 | |
| *** yamamoto has quit IRC | 01:39 | |
| *** yamamoto has joined #openstack-meeting-3 | 01:40 | |
| *** yamamoto has quit IRC | 01:41 | |
| *** yamamoto has joined #openstack-meeting-3 | 01:42 | |
| *** yamamoto has quit IRC | 01:50 | |
| *** ianychoi has quit IRC | 03:00 | |
| *** ianychoi has joined #openstack-meeting-3 | 03:21 | |
| *** ralonsoh has joined #openstack-meeting-3 | 07:41 | |
| *** Luzi has joined #openstack-meeting-3 | 07:45 | |
| *** slaweq has joined #openstack-meeting-3 | 08:01 | |
| *** e0ne has joined #openstack-meeting-3 | 08:10 | |
| *** e0ne has quit IRC | 08:12 | |
| *** macz_ has joined #openstack-meeting-3 | 08:18 | |
| *** macz_ has quit IRC | 08:23 | |
| *** lpetrut has joined #openstack-meeting-3 | 08:36 | |
| *** tosky has joined #openstack-meeting-3 | 08:40 | |
| *** lyarwood has quit IRC | 09:01 | |
| *** e0ne has joined #openstack-meeting-3 | 09:10 | |
| *** e0ne has quit IRC | 09:10 | |
| *** macz_ has joined #openstack-meeting-3 | 10:06 | |
| *** macz_ has quit IRC | 10:11 | |
| *** Luzi has quit IRC | 10:55 | |
| *** macz_ has joined #openstack-meeting-3 | 10:57 | |
| *** e0ne has joined #openstack-meeting-3 | 10:59 | |
| *** macz_ has quit IRC | 11:02 | |
| *** raildo has joined #openstack-meeting-3 | 12:25 | |
| *** Luzi has joined #openstack-meeting-3 | 12:56 | |
| *** e0ne has quit IRC | 13:02 | |
| *** lpetrut has quit IRC | 13:02 | |
| *** e0ne has joined #openstack-meeting-3 | 13:27 | |
| *** e0ne has quit IRC | 13:29 | |
| *** e0ne has joined #openstack-meeting-3 | 13:38 | |
| *** slaweq has quit IRC | 13:42 | |
| *** slaweq has joined #openstack-meeting-3 | 13:47 | |
| *** mlavalle has joined #openstack-meeting-3 | 13:59 | |
| *** obondarev has joined #openstack-meeting-3 | 13:59 | |
| *** lajoskatona has joined #openstack-meeting-3 | 14:00 | |
| slaweq | #startmeeting networking | 14:00 |
|---|---|---|
| openstack | Meeting started Tue Nov 3 14:00:43 2020 UTC and is due to finish in 60 minutes. The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot. | 14:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 14:00 |
| *** openstack changes topic to " (Meeting topic: networking)" | 14:00 | |
| openstack | The meeting name has been set to 'networking' | 14:00 |
| slaweq | hi | 14:00 |
| mlavalle | o/ | 14:00 |
| obondarev | o/ | 14:00 |
| ralonsoh | hi | 14:00 |
| bcafarel | o/ | 14:01 |
| lajoskatona | o/ | 14:01 |
| rubasov | o/ | 14:02 |
| slaweq | ok, lets start | 14:02 |
| slaweq | #topic Announcements | 14:03 |
| *** openstack changes topic to "Announcements (Meeting topic: networking)" | 14:03 | |
| slaweq | I fast-tracked back Oleg to the core team - welcome back in the team obondarev :) | 14:03 |
| obondarev | Thanks!!! :) | 14:03 |
| ralonsoh | welcome | 14:04 |
| obondarev | thanks folks, I'm happy to join the team | 14:05 |
| slaweq | next one | 14:05 |
| slaweq | I proposed Rodolfo to be member of the stable core team: http://lists.openstack.org/pipermail/openstack-discuss/2020-November/018480.html | 14:05 |
| mlavalle | obondarev: WELCOME! | 14:05 |
| slaweq | I hope it will be done this week | 14:05 |
| ralonsoh | I'll be able to break stable branches too | 14:06 |
| slaweq | LOL | 14:06 |
| slaweq | true | 14:06 |
| lajoskatona | LOL.... | 14:06 |
| slaweq | next one | 14:06 |
| slaweq | Virtual PTG summary http://lists.openstack.org/pipermail/openstack-discuss/2020-November/018489.html | 14:06 |
| slaweq | also on http://kaplonski.pl/blog/virtual_ptg_october_2020_summary/ where there are also team photos | 14:06 |
| slaweq | and regarding ptg agreements | 14:07 |
| slaweq | QoS meeting is now cancelled - https://review.opendev.org/#/c/760902/ - if we will have any QoS related topics, please add them to the On demand agenda for the team meeting | 14:07 |
| slaweq | also | 14:09 |
| slaweq | CI meeting time changed - now it will be on Tuesday, just after this meeting: https://review.opendev.org/#/c/760823/ | 14:09 |
| slaweq | next one | 14:09 |
| bcafarel | congrats obondarev and ralonsoh (and watch out ralonsoh I will be watching :) ) | 14:09 |
| slaweq | Wallaby cycle calendar https://releases.openstack.org/wallaby/schedule.html | 14:09 |
| njohnston | o/ | 14:09 |
| bcafarel | (sorry for lag, dns issues) | 14:09 |
| slaweq | it is our new most important calendar for now :) | 14:10 |
| slaweq | First milestone is in the week of November 30th | 14:10 |
| slaweq | so we have just few weeks to that milestone | 14:10 |
| slaweq | and last one for today | 14:10 |
| slaweq | gerrit outage is planned Nov 20 - 23 - http://lists.opendev.org/pipermail/service-announce/2020-October/000012.html | 14:11 |
| slaweq | just FYI if You missed this email on openinfra ML | 14:11 |
| slaweq | that's all announcements for today from my side | 14:11 |
| slaweq | anything else You want to share with the team now? | 14:12 |
| bcafarel | stable-wise, stein is going EM next week | 14:12 |
| bcafarel | we will tag one last neutron version for it before that, if you want to have a backport in a tag, better hurry | 14:12 |
| bcafarel | of course we will still do backports after that, just that there won't be any new tag releases | 14:13 |
| slaweq | bcafarel: will we do last release this week? | 14:13 |
| bcafarel | https://review.opendev.org/#/q/status:open+project:openstack/neutron+branch:stable/stein is almost empty so we can prepare it on Thursday to give some time? | 14:14 |
| slaweq | ++ | 14:15 |
| slaweq | ok, lets move on | 14:16 |
| slaweq | next topic | 14:16 |
| slaweq | #topic Blueprints | 14:16 |
| *** openstack changes topic to "Blueprints (Meeting topic: networking)" | 14:16 | |
| slaweq | I prepared some list of the BPs for W-1 https://bugs.launchpad.net/neutron/+milestone/wallaby-1 | 14:16 |
| slaweq | those are mostly things which we postponed from last cycle | 14:16 |
| slaweq | and which I think that will be continued now | 14:17 |
| slaweq | if You want to add some other BP to that list and track it weekly, please tell me that on irc | 14:17 |
| slaweq | also if You think we should move out something from that list, please mee that too | 14:17 |
| slaweq | we need some volunteer for https://blueprints.launchpad.net/neutron/+spec/handle-deadlocks-in-oslo-way - at least someone who will organize work on that | 14:20 |
| slaweq | as this seems to be long one | 14:20 |
| slaweq | similar to e.g. new engine facade transition | 14:20 |
| lajoskatona | parttime I can work on that | 14:21 |
| *** jlibosva has joined #openstack-meeting-3 | 14:21 | |
| slaweq | lajoskatona: thx a lot | 14:21 |
| slaweq | lajoskatona: also question to You about https://bugs.launchpad.net/bugs/1882804 | 14:22 |
| openstack | Launchpad bug 1882804 in neutron "RFE: allow replacing the QoS policy of bound port" [Wishlist,Confirmed] - Assigned to Lajos Katona (lajos-katona) | 14:22 |
| slaweq | what we still need there? | 14:22 |
| lajoskatona | I think only the tempest test, but let me check | 14:22 |
| slaweq | IIRC there was some bug in neutron code which broke nova's gate | 14:22 |
| slaweq | and that's why we didn't finish it in Victoria | 14:22 |
| slaweq | right? | 14:22 |
| lajoskatona | yeah the fix isn't merged: https://review.opendev.org/756892 | 14:23 |
| lajoskatona | we postponed to Wallaby | 14:24 |
| slaweq | and after that what else we will need to finish that? | 14:25 |
| lajoskatona | Tempest: https://review.opendev.org/743695 | 14:25 |
| lajoskatona | the documentation was updated, so I think nothing is missing except the bugfix (https://review.opendev.org/756892 ) | 14:26 |
| slaweq | ok, so seems like that one is close | 14:26 |
| slaweq | thx for update lajoskatona | 14:27 |
| obondarev | I've just +A bug fix | 14:27 |
| obondarev | :) | 14:27 |
| slaweq | thx obondarev | 14:27 |
| obondarev | (it had +1 from me) | 14:27 |
| obondarev | so not just blind +A :D | 14:28 |
| lajoskatona | thanks for the attention and reviews | 14:28 |
| slaweq | ok, any other updates about BPs? | 14:28 |
| slaweq | ok, so lets move on | 14:29 |
| slaweq | #topic Community Goals | 14:29 |
| *** openstack changes topic to "Community Goals (Meeting topic: networking)" | 14:29 | |
| slaweq | we have now one community goal: | 14:30 |
| slaweq | Migrate from oslo.rootwrap to oslo.privsep | 14:30 |
| slaweq | https://governance.openstack.org/tc/goals/selected/wallaby/migrate-to-privsep.html | 14:30 |
| lajoskatona | and the policy.yaml topic was not accepted for this cycle? | 14:30 |
| amotoki | it will be accepted soon in my understanding. | 14:30 |
| slaweq | lajoskatona: https://governance.openstack.org/tc/goals/selected/wallaby/ | 14:30 |
| slaweq | for now I see only that one | 14:31 |
| lajoskatona | it's still not merged: https://review.opendev.org/759881, so I am not sure if it will be wallaby goal | 14:31 |
| lajoskatona | And the etherpad mentions as goal: https://etherpad.opendev.org/p/policy-popup-wallaby-ptg but not sure | 14:32 |
| slaweq | thx lajoskatona and amotoki for the heads up | 14:33 |
| amotoki | the proposed one is about changing the default policy file name to policy.yaml. Other policy stuffs are not part of the current proposed one. | 14:33 |
| slaweq | I will add it to our agenda | 14:33 |
| amotoki | anyway I will care it. | 14:34 |
| slaweq | even if it's not yet accepted officially we can try to work on it | 14:34 |
| slaweq | amotoki: thx a lot | 14:34 |
| slaweq | I think we can move on to the next topic now | 14:35 |
| slaweq | #topic Bugs | 14:35 |
| *** openstack changes topic to "Bugs (Meeting topic: networking)" | 14:35 | |
| rubasov | report: http://lists.openstack.org/pipermail/openstack-discuss/2020-November/018494.html | 14:35 |
| slaweq | rubasov: thx | 14:35 |
| rubasov | sure | 14:35 |
| slaweq | (that was fast :)) | 14:35 |
| rubasov | it was quite a slow week | 14:35 |
| rubasov | however we have some incomplete bugs | 14:36 |
| rubasov | first let me call your attention to this one | 14:36 |
| rubasov | https://bugs.launchpad.net/neutron/+bug/1901707 | 14:36 |
| openstack | Launchpad bug 1901707 in neutron "race condition on port binding vs instance being resumed for live-migrations" [Undecided,Incomplete] | 14:36 |
| rubasov | which may not be incomplete to someone who understands the history of the issue in depth | 14:37 |
| rubasov | but I myself had to ask a few questions from the reporter, so marked it incomplete | 14:37 |
| ralonsoh | we found this bug too in our CI | 14:38 |
| ralonsoh | the problem is the load of the OVS agent | 14:38 |
| rubasov | oh, that's a piece of information I did not encounter while reading the history | 14:38 |
| rubasov | the problem seems relevant (failed connectivity after some live migrations) | 14:39 |
| rubasov | but it would be great if we could simplify the issue | 14:40 |
| ralonsoh | but that should be solved with the new port binding method | 14:40 |
| ralonsoh | implemented in Rocky+ | 14:40 |
| rubasov | the old bug report (https://bugs.launchpad.net/neutron/+bug/1815989) explicitly states the opposite IIRC | 14:41 |
| openstack | Launchpad bug 1815989 in OpenStack Compute (nova) "OVS drops RARP packets by QEMU upon live-migration causes up to 40s ping pause in Rocky" [Medium,In progress] - Assigned to sean mooney (sean-k-mooney) | 14:41 |
| ralonsoh | he is still working on this | 14:41 |
| ralonsoh | I know that for sure | 14:41 |
| slaweq | so should we mark 1901707 as a duplicate of 1815989 ? | 14:42 |
| slaweq | or is it different issue? | 14:43 |
| rubasov | the reporter's intention was to break out a sub-problem from the old and huge report | 14:43 |
| rubasov | but I did not fully manage to understand to boundaries between the two parts | 14:43 |
| ralonsoh | ok, let's keep both for now | 14:44 |
| ralonsoh | I'm working with Sean in the original one | 14:44 |
| slaweq | ralonsoh: ok, so please keep an eye on that new one too | 14:44 |
| *** macz_ has joined #openstack-meeting-3 | 14:44 | |
| rubasov | okay that tells me practically we have progress on the 2nd report too | 14:44 |
| rubasov | thank you | 14:44 |
| rubasov | another is this: https://bugs.launchpad.net/neutron/+bug/1902211 | 14:45 |
| openstack | Launchpad bug 1902211 in neutron "Router State standby on all l3 agent when create" [Undecided,Incomplete] | 14:45 |
| rubasov | if you have experience with debugging locks not released and vpnaas, the reporter could use your help | 14:46 |
| rubasov | but it's far from simple, because the bug is not reproducible at will | 14:46 |
| rubasov | these were the bugs I wanted to mention | 14:48 |
| rubasov | the rest I believe has progress | 14:49 |
| *** macz_ has quit IRC | 14:49 | |
| slaweq | so seems like some neutron-vpnaas bug? | 14:50 |
| rubasov | could easily be | 14:50 |
| slaweq | I just wrote a comment there | 14:52 |
| slaweq | thx rubasov for the summary | 14:52 |
| *** purplerbot has quit IRC | 14:52 | |
| slaweq | this week our bug deputy is ralonsoh | 14:52 |
| slaweq | and next week will be lucasgomes' turn | 14:53 |
| ralonsoh | on my way! | 14:53 |
| slaweq | I also added obondarev to our bug deputy rotation calendar now :) | 14:53 |
| obondarev | np :) | 14:53 |
| slaweq | any other bugs You would like to quickly discuss today? | 14:54 |
| slaweq | ok, I guess that this means "no" | 14:56 |
| slaweq | please remember - in 4 minutes we have ci meeting in same channel :) | 14:56 |
| slaweq | thx for attending that meeting and see You online | 14:56 |
| slaweq | o/ | 14:56 |
| slaweq | #endmeeting | 14:56 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 14:56 | |
| openstack | Meeting ended Tue Nov 3 14:56:41 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 14:56 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/networking/2020/networking.2020-11-03-14.00.html | 14:56 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/networking/2020/networking.2020-11-03-14.00.txt | 14:56 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/networking/2020/networking.2020-11-03-14.00.log.html | 14:56 |
| amotoki | o/ | 14:56 |
| lajoskatona | Bye | 14:56 |
| rubasov | bye | 14:57 |
| mlavalle | o/ | 14:57 |
| * bcafarel updates his calendar | 14:58 | |
| slaweq | #startmeeting neutron_ci | 15:00 |
| openstack | Meeting started Tue Nov 3 15:00:10 2020 UTC and is due to finish in 60 minutes. The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| *** openstack changes topic to " (Meeting topic: neutron_ci)" | 15:00 | |
| slaweq | welcome back :) | 15:00 |
| openstack | The meeting name has been set to 'neutron_ci' | 15:00 |
| bcafarel | long time no see :) | 15:00 |
| ralonsoh | hi | 15:01 |
| slaweq | bcafarel: yeah :D | 15:01 |
| obondarev | o/ | 15:01 |
| mlavalle | o/ | 15:01 |
| slaweq | Grafana dashboard: http://grafana.openstack.org/dashboard/db/neutron-failure-rate | 15:02 |
| slaweq | lets open it now and we can start | 15:02 |
| slaweq | #topic Actions from previous meetings | 15:02 |
| *** openstack changes topic to "Actions from previous meetings (Meeting topic: neutron_ci)" | 15:02 | |
| slaweq | slaweq to propose patch to check console log before ssh to instance | 15:02 |
| slaweq | Done: https://review.opendev.org/#/c/758968/ | 15:02 |
| ralonsoh | +1 to this patch | 15:03 |
| slaweq | and TBH I didn't saw AuthenticationFailure errors in neutron-tempest-plugin jobs in last few days | 15:03 |
| slaweq | so it seems that it could helps really | 15:03 |
| ralonsoh | until we find/fix the error in paramiko, that will help | 15:03 |
| slaweq | I will try to do something similar to tempest also | 15:03 |
| slaweq | #action slaweq to propose patch to check console log before ssh to instance in tempest | 15:04 |
| slaweq | next one | 15:04 |
| slaweq | bcafarel to update grafana dashboard for master branch | 15:04 |
| bcafarel | not merged yet, but has a +2 https://review.opendev.org/#/c/758208/ | 15:05 |
| slaweq | thx | 15:05 |
| slaweq | ok, last one from previous meeting | 15:06 |
| slaweq | slaweq to check failing neutron-grenade-ovn job | 15:06 |
| slaweq | I didn't had time for that still | 15:06 |
| slaweq | #action slaweq to check failing neutron-grenade-ovn job | 15:06 |
| slaweq | and that's all actions from last week | 15:06 |
| slaweq | lets move on | 15:06 |
| slaweq | #topic Stadium projects | 15:06 |
| *** openstack changes topic to "Stadium projects (Meeting topic: neutron_ci)" | 15:06 | |
| slaweq | lajoskatona: anything regarding stadium projects and ci? | 15:06 |
| lajoskatona | Hi | 15:07 |
| lajoskatona | nothing new | 15:07 |
| lajoskatona | I still in the recovering phase after PTG, sorry | 15:07 |
| slaweq | AFAICT for stadium projects it is pretty stable, at least I didn't saw many failures | 15:07 |
| lajoskatona | yeah the problems appear mostly in older branches | 15:08 |
| bcafarel | for which I have a PTG action item I think :) | 15:08 |
| slaweq | yeah, to check which ones are broken and should be moved to "unmaintained" phase | 15:08 |
| slaweq | :) | 15:08 |
| slaweq | btw. there is one thing regarding stadium, mlavalle please check https://review.opendev.org/#/q/topic:neutron-victoria+(status:open+OR+status:merged) | 15:09 |
| slaweq | those are patches for stable/victoria | 15:09 |
| *** Luzi has quit IRC | 15:10 | |
| slaweq | we need to switch there to use neutron-tempest-plugin-victoria jobs | 15:10 |
| slaweq | if there is nothing more related to the stadium, lets move on | 15:10 |
| slaweq | next topic | 15:11 |
| slaweq | #topic Stable branches | 15:11 |
| *** openstack changes topic to "Stable branches (Meeting topic: neutron_ci)" | 15:11 | |
| slaweq | Victoria dashboard: http://grafana.openstack.org/d/pM54U-Kiz/neutron-failure-rate-previous-stable-release?orgId=1 | 15:11 |
| slaweq | Ussuri dashboard: http://grafana.openstack.org/d/dCFVU-Kik/neutron-failure-rate-older-stable-release?orgId=1 | 15:11 |
| mlavalle | there are patches for master also in that url | 15:11 |
| mlavalle | or am I misunderstanding? | 15:11 |
| slaweq | mlavalle: no, only for stable/victoria | 15:11 |
| slaweq | in master branch we are still using base neutron-tempest-plugin jobs | 15:12 |
| slaweq | but for stable/victoria we need to run jobs dedicated for stable/victoria | 15:12 |
| mlavalle | ok, I think I clicked it wrong | 15:12 |
| slaweq | :) | 15:12 |
| slaweq | bcafarel: any new issues with stable branches? | 15:13 |
| bcafarel | nothing I spotted this week, hopefully we have rocky/queens back on track now thanks to your patch | 15:14 |
| slaweq | yes, this should be better with https://review.opendev.org/#/c/758377/ :) | 15:14 |
| slaweq | ok, so lets move on | 15:15 |
| slaweq | #topic Grafana | 15:15 |
| *** openstack changes topic to "Grafana (Meeting topic: neutron_ci)" | 15:15 | |
| slaweq | #link http://grafana.openstack.org/d/Hj5IHcSmz/neutron-failure-rate?orgId=1 | 15:15 |
| lajoskatona | I have to leave now, perhaps I can join later (in 30minutes) if I find wifi to connect.... | 15:16 |
| *** lajoskatona has quit IRC | 15:16 | |
| slaweq | still I think that most failing jobs are (non-voting) ovn related jobs | 15:17 |
| slaweq | like e.g. http://grafana.openstack.org/d/Hj5IHcSmz/neutron-failure-rate?viewPanel=18&orgId=1 | 15:18 |
| slaweq | is there any volunteer who will want to check those failures? | 15:18 |
| slaweq | jlibosva ? | 15:18 |
| slaweq | :) | 15:18 |
| jlibosva | I can put it on my todo list :) | 15:19 |
| slaweq | jlibosva: thx | 15:20 |
| bcafarel | so we have at least https://bugs.launchpad.net/neutron/+bug/1902512 | 15:20 |
| openstack | Launchpad bug 1902512 in neutron "neutron-ovn-tripleo-ci-centos-8-containers-multinode fails on private networ creation (mtu size)" [Medium,Triaged] | 15:20 |
| slaweq | yes, this one sounds like serious one because it happens often | 15:21 |
| slaweq | but I think that in other, devstack based jobs there are other failures | 15:21 |
| slaweq | ok, except that I think it's "normal" on grafana | 15:22 |
| slaweq | so we can move on to the specific jobs and failures | 15:22 |
| slaweq | ok? | 15:22 |
| slaweq | #topic fullstack/functional | 15:23 |
| *** openstack changes topic to "fullstack/functional (Meeting topic: neutron_ci)" | 15:23 | |
| bcafarel | sounds good | 15:23 |
| slaweq | here I found couple of new issues | 15:24 |
| slaweq | first functional tests | 15:24 |
| slaweq | I (again) so job timeout due to high amount of logs: | 15:24 |
| slaweq | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_6dc/755752/3/check/neutron-functional-with-uwsgi/6dcca4f/job-output.txt | 15:24 |
| slaweq | it's old issue with stestr iirc | 15:24 |
| slaweq | I will open LP for that today | 15:25 |
| slaweq | any volunteer to take a look and maybe try to avoid some logs to be send to stdout? | 15:25 |
| ralonsoh | not this week, sorry | 15:25 |
| slaweq | ok, I will open LP and if someone will have time, You can take it :) | 15:26 |
| slaweq | now fulstack | 15:26 |
| slaweq | I reported bug https://bugs.launchpad.net/neutron/+bug/1902678 today | 15:27 |
| openstack | Launchpad bug 1902678 in neutron "[Fullstack] Wrong output of the cmdline causes tests timeouts" [Critical,Confirmed] | 15:27 |
| slaweq | I saw it at least 3 times recently | 15:27 |
| slaweq | basically if that happens, it will fail many tests as in all of them it will timeout while waiting until dhcp agent process will be spawned | 15:27 |
| slaweq | it looks like in https://zuul.opendev.org/t/openstack/build/40affb0d6e0844369a293b05dea0e42c/log/controller/logs/dsvm-fullstack-logs/TestHAL3Agent.test_gateway_ip_changed.txt | 15:28 |
| slaweq | anyone interested in checking that? | 15:29 |
| ralonsoh | ok, I'll take a look | 15:30 |
| slaweq | thx | 15:30 |
| slaweq | #action ralonsoh to check fullstack issue https://bugs.launchpad.net/neutron/+bug/1902678 | 15:31 |
| openstack | Launchpad bug 1902678 in neutron "[Fullstack] Wrong output of the cmdline causes tests timeouts" [Critical,Confirmed] | 15:31 |
| slaweq | ok, lets move on to the scenario jobs now | 15:31 |
| slaweq | #topic Tempest/Scenario | 15:31 |
| *** openstack changes topic to "Tempest/Scenario (Meeting topic: neutron_ci)" | 15:31 | |
| slaweq | first issue which I saw few times are problems with cinder volumes | 15:32 |
| slaweq | like e.g. https://2f507ad644729ed0a17c-1abd4c4163ab8d95786215227f5e857f.ssl.cf5.rackcdn.com/758098/7/check/tempest-slow-py3/b2b284b/testr_results.html or https://zuul.opendev.org/t/openstack/build/eec3c390c2944d0ab56460c75d0383fa/logs | 15:32 |
| slaweq | and I was thinking about maybe blacklisting those failing cinder tests in our jobs? | 15:32 |
| slaweq | wdyt about it? | 15:32 |
| bcafarel | can it be done easily in zuul? this is a global job definition no? | 15:33 |
| slaweq | bcafarel: tempest-slow-py3 is defined in tempest repo | 15:33 |
| bcafarel | (if doable definitely +1 with "to restore once cinder is fixed") | 15:33 |
| bcafarel | oh nice | 15:34 |
| slaweq | but neutron-tempest-multinode-full-py3 is defined in neutron | 15:34 |
| slaweq | but for tempest-slow-py3 we can do our job "neutron-tempest-slow-py3" and blacklist such tests there | 15:34 |
| slaweq | I don't know if gmann will be happy with that if he will discover it but we can try IMO ;) | 15:34 |
| gmann | slaweq: you mean do neutron-tempest-slow-py3 like we did for integrated job? | 15:35 |
| slaweq | gmann: yes, something like that | 15:36 |
| slaweq | but also without "volume" tests which | 15:36 |
| slaweq | which are failing pretty often in our jobs | 15:36 |
| gmann | i think that make sense, I will say we did not do it for slow/multinode job but we should do | 15:36 |
| slaweq | and we are trying to make our CI a bit more stable because now it is a nightmare | 15:36 |
| gmann | +1 | 15:37 |
| slaweq | thx :) | 15:37 |
| slaweq | ok, so I will do it in our repo | 15:37 |
| slaweq | #action slaweq to blacklist some cinder related tests in the neutron-tempest-* jobs | 15:38 |
| gmann | either is fine, i think it will be used in neutron so neutron-tempest-slow-py in neutron repo make sense | 15:38 |
| gmann | if it is more than neutron then we can do in tempest repo | 15:38 |
| slaweq | gmann: ok | 15:38 |
| slaweq | ok, lets move to the grenade jobs now | 15:40 |
| slaweq | and with grenade jobs I have one "issue" but maybe it's just my missunderstanding of something | 15:40 |
| slaweq | it seems that in multinode grenade jobs services on compute-1 node aren't upgraded | 15:41 |
| slaweq | and that is causing failure with unsupported ovo version in e.g. my patch https://9a7a3a32fbdea177beae-de1ec222256e01db8c1f1f4d7a4b9170.ssl.cf5.rackcdn.com/749158/8/check/neutron-grenade-multinode/a45a7ed/compute1/logs/screen-neutron-agent.txt | 15:41 |
| slaweq | now the question is - should it be like that and we should be able to run compute node with older agents | 15:41 |
| slaweq | or should we upgrade those agents too? | 15:42 |
| slaweq | do You know? | 15:42 |
| bcafarel | hmm newer controller and older compute should work no? | 15:42 |
| bcafarel | though in grenade I expected a full upgrade | 15:43 |
| *** obondarev has quit IRC | 15:43 | |
| slaweq | ok, so I will investigate why it's not working if it should | 15:44 |
| slaweq | gmann: also, I found out recently that in grenade jobs on subnodes we are using lib/neutron instead of lib/neutron-legacy | 15:44 |
| slaweq | gmann: can You check https://review.opendev.org/#/c/759199/ maybe? | 15:44 |
| gmann | slaweq: sure | 15:44 |
| slaweq | thx | 15:44 |
| slaweq | ok, and that's basically all from me for today | 15:45 |
| slaweq | please remember to check failed jobs before recheck | 15:45 |
| mlavalle | slaweq: I think hangyan had a similar issue with one of his patches and grenade | 15:45 |
| slaweq | and write related bug (or open new one if needed) while rechecking | 15:45 |
| slaweq | mlavalle: yes, I know | 15:46 |
| slaweq | but I will check on my patch while it's like, maybe we are doing something wrong there :) | 15:46 |
| mlavalle | ok | 15:46 |
| mlavalle | I'll mention this to him | 15:46 |
| slaweq | thx | 15:47 |
| slaweq | ok, if there is nothing else to be discussed today, I will give You few minutes back | 15:48 |
| slaweq | thx for attending the meeting | 15:48 |
| slaweq | and see You online | 15:48 |
| ralonsoh | bye | 15:48 |
| slaweq | o/ | 15:48 |
| slaweq | #endmeeting | 15:48 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 15:48 | |
| openstack | Meeting ended Tue Nov 3 15:48:32 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:48 |
| mlavalle | o/ | 15:48 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-11-03-15.00.html | 15:48 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-11-03-15.00.txt | 15:48 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-11-03-15.00.log.html | 15:48 |
| *** mlavalle has left #openstack-meeting-3 | 15:48 | |
| *** bnemec has quit IRC | 15:50 | |
| bcafarel | o/ | 15:52 |
| *** lajoskatona has joined #openstack-meeting-3 | 15:59 | |
| *** macz_ has joined #openstack-meeting-3 | 16:18 | |
| *** lajoskatona has quit IRC | 16:50 | |
| *** jlibosva has quit IRC | 16:59 | |
| *** bnemec has joined #openstack-meeting-3 | 17:00 | |
| *** lpetrut has joined #openstack-meeting-3 | 17:02 | |
| *** purplerbot has joined #openstack-meeting-3 | 17:03 | |
| *** e0ne has quit IRC | 17:20 | |
| *** tosky has quit IRC | 17:57 | |
| *** e0ne has joined #openstack-meeting-3 | 18:06 | |
| *** lajoskatona has joined #openstack-meeting-3 | 18:24 | |
| *** lajoskatona has left #openstack-meeting-3 | 18:26 | |
| *** lpetrut has quit IRC | 18:26 | |
| *** ralonsoh has quit IRC | 18:49 | |
| *** e0ne has quit IRC | 18:52 | |
| *** e0ne has joined #openstack-meeting-3 | 20:12 | |
| *** tosky has joined #openstack-meeting-3 | 20:27 | |
| *** slaweq has quit IRC | 20:33 | |
| *** e0ne has quit IRC | 20:35 | |
| *** slaweq has joined #openstack-meeting-3 | 20:44 | |
| *** raildo has quit IRC | 21:35 | |
| *** e0ne has joined #openstack-meeting-3 | 21:58 | |
| *** slaweq has quit IRC | 22:02 | |
| *** mlavalle has joined #openstack-meeting-3 | 22:11 | |
| *** e0ne has quit IRC | 22:34 | |
| *** tosky has quit IRC | 23:01 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!