| *** tosky has quit IRC | 00:06 | |
| *** yamamoto has joined #openstack-meeting-3 | 00:24 | |
| *** macz_ has quit IRC | 00:56 | |
| *** baojg has joined #openstack-meeting-3 | 01:04 | |
| *** yamamoto has quit IRC | 01:09 | |
| *** mlavalle has quit IRC | 01:10 | |
| *** yamamoto has joined #openstack-meeting-3 | 01:23 | |
| *** artom has quit IRC | 02:55 | |
| *** njohnston has quit IRC | 06:03 | |
| *** yamamoto has quit IRC | 07:07 | |
| *** yamamoto has joined #openstack-meeting-3 | 07:40 | |
| *** ralonsoh has joined #openstack-meeting-3 | 07:43 | |
| *** yamamoto has quit IRC | 07:45 | |
| *** yamamoto has joined #openstack-meeting-3 | 07:54 | |
| *** macz_ has joined #openstack-meeting-3 | 07:57 | |
| *** slaweq has joined #openstack-meeting-3 | 07:58 | |
| *** macz_ has quit IRC | 08:02 | |
| *** tosky has joined #openstack-meeting-3 | 08:56 | |
| *** artom has joined #openstack-meeting-3 | 10:19 | |
| *** yamamoto has quit IRC | 10:29 | |
| *** e0ne has joined #openstack-meeting-3 | 11:08 | |
| *** baojg has quit IRC | 11:10 | |
| *** baojg has joined #openstack-meeting-3 | 11:12 | |
| *** baojg has quit IRC | 11:21 | |
| *** yamamoto has joined #openstack-meeting-3 | 11:30 | |
| *** macz_ has joined #openstack-meeting-3 | 11:33 | |
| *** macz_ has quit IRC | 11:38 | |
| *** yamamoto has quit IRC | 11:45 | |
| *** slaweq has quit IRC | 11:56 | |
| *** slaweq has joined #openstack-meeting-3 | 11:57 | |
| *** yamamoto has joined #openstack-meeting-3 | 12:02 | |
| *** yamamoto has quit IRC | 12:10 | |
| *** ianychoi_ has quit IRC | 12:10 | |
| *** raildo_ has joined #openstack-meeting-3 | 12:14 | |
| *** raildo has quit IRC | 12:16 | |
| *** yamamoto has joined #openstack-meeting-3 | 12:24 | |
| *** yamamoto has quit IRC | 12:24 | |
| *** yamamoto has joined #openstack-meeting-3 | 12:24 | |
| *** lpetrut has joined #openstack-meeting-3 | 12:24 | |
| *** yamamoto has quit IRC | 12:29 | |
| *** Luzi has joined #openstack-meeting-3 | 12:36 | |
| *** njohnston_ has joined #openstack-meeting-3 | 12:43 | |
| *** artom has quit IRC | 13:18 | |
| *** yamamoto has joined #openstack-meeting-3 | 13:39 | |
| *** mlavalle has joined #openstack-meeting-3 | 13:56 | |
| *** obondarev has joined #openstack-meeting-3 | 13:58 | |
| *** baojg has joined #openstack-meeting-3 | 14:00 | |
| slaweq | #startmeeting networking | 14:00 |
|---|---|---|
| openstack | Meeting started Tue Nov 17 14:00:13 2020 UTC and is due to finish in 60 minutes. The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot. | 14:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 14:00 |
| *** openstack changes topic to " (Meeting topic: networking)" | 14:00 | |
| openstack | The meeting name has been set to 'networking' | 14:00 |
| njohnston_ | o/ | 14:00 |
| *** njohnston_ is now known as njohnston | 14:00 | |
| mlavalle | o/ | 14:00 |
| slaweq | o/ | 14:00 |
| bcafarel | o/ | 14:01 |
| obondarev | hi | 14:01 |
| rubasov | o/ | 14:01 |
| haleyb | hi | 14:02 |
| slaweq | #topic Announcements | 14:03 |
| *** openstack changes topic to "Announcements (Meeting topic: networking)" | 14:03 | |
| slaweq | I don't have many announcements for today | 14:03 |
| slaweq | just 2 reminders | 14:03 |
| slaweq | Wallaby-1 milestone is just in 2 weeks so we are close to it | 14:03 |
| slaweq | and second | 14:03 |
| slaweq | this weekend gerrit outage is planned: http://lists.opendev.org/pipermail/service-announce/2020-October/000012.html | 14:04 |
| slaweq | please be aware of it if You want to work on something during those days ;) | 14:04 |
| slaweq | anything else You want to share with the team? | 14:04 |
| ralonsoh | (sorry I'm in other meeting) | 14:05 |
| slaweq | ok, I guess that this means "no" :) | 14:06 |
| slaweq | and we can move on | 14:06 |
| slaweq | #topic Blueprints | 14:07 |
| *** openstack changes topic to "Blueprints (Meeting topic: networking)" | 14:07 | |
| slaweq | Wallaby-1 https://bugs.launchpad.net/neutron/+milestone/wallaby-1 | 14:07 |
| slaweq | any updates about any of BPs? | 14:07 |
| mlavalle | We continue working on address groups. Couple of patches will be coming soon | 14:07 |
| slaweq | mlavalle: and one of them is actually blocked by https://review.opendev.org/#/c/762434/ | 14:08 |
| slaweq | right? | 14:08 |
| mlavalle | That's a third one | 14:09 |
| mlavalle | the API and DB patch for address groups in security group rules | 14:09 |
| slaweq | I will check obondarev's comments there later today and will try also to check why it's still not working as it should | 14:09 |
| slaweq | so hopefully it will be ready this week | 14:09 |
| mlavalle | Thanks! | 14:09 |
| slaweq | any other updates? | 14:10 |
| slaweq | ok | 14:11 |
| slaweq | I have 2 quick updates | 14:11 |
| slaweq | 1. I closed BP related to the db retries as that's what we decided on the last drivers meeting | 14:13 |
| slaweq | https://blueprints.launchpad.net/neutron/+spec/handle-deadlocks-in-oslo-way | 14:13 |
| *** lajoskatona has joined #openstack-meeting-3 | 14:13 | |
| lajoskatona | Hi | 14:14 |
| slaweq | 2. I was pinged today by xinranwang who asked me about review of https://review.opendev.org/#/c/742785/ | 14:14 |
| slaweq | it is nova spec, but it has also some parts related to the neutron | 14:14 |
| * bcafarel notes to read last drivers' meeting log | 14:14 | |
| slaweq | that's at least what he told me | 14:14 |
| slaweq | so, please add this spec to Your review list and check it | 14:15 |
| slaweq | and that's basically all from my side regarding BPs | 14:16 |
| slaweq | if there are no other updates, lets move on | 14:17 |
| slaweq | #topic Community Goals | 14:17 |
| *** openstack changes topic to "Community Goals (Meeting topic: networking)" | 14:17 | |
| slaweq | Migrate from oslo.rootwrap to oslo.privsep driven by ralonsoh | 14:17 |
| slaweq | I started preparing some list of changes which we need to do here to accomplish that | 14:18 |
| *** Luzi has quit IRC | 14:18 | |
| slaweq | I didn't finish it yet | 14:18 |
| slaweq | but I have a question to all of You | 14:18 |
| slaweq | I found in neutron tree module: neutron/agent/linux/xenapi_root_helper.py | 14:18 |
| slaweq | my question is: do we still should keep it there? is it used by anyone? | 14:18 |
| slaweq | I don't think we are doing any kind of testing of it really | 14:19 |
| ralonsoh | it was removed in Nova: https://review.opendev.org/#/c/749309/, https://review.opendev.org/#/c/749304/ | 14:19 |
| slaweq | thx ralonsoh | 14:19 |
| *** macz_ has joined #openstack-meeting-3 | 14:20 | |
| ralonsoh | I think we should mark the conf options as deprecated and remove all the code | 14:20 |
| ralonsoh | and in X, remove the conf options | 14:20 |
| njohnston | +1 | 14:20 |
| bcafarel | cleanup++ | 14:21 |
| obondarev | +1 | 14:22 |
| slaweq | do You think that we should first send email to ML to ensure that nobody is using it really? | 14:23 |
| lajoskatona | +1 | 14:24 |
| ralonsoh | I don't think so | 14:24 |
| *** macz_ has quit IRC | 14:24 | |
| ralonsoh | that was sent 1 year ago | 14:24 |
| njohnston | personally I think we can go by the due diligence nova would have done before deprecating their part | 14:25 |
| ralonsoh | then Nova marked it as deprecated | 14:25 |
| ralonsoh | in 2019 | 14:25 |
| slaweq | ok | 14:25 |
| slaweq | great :) | 14:25 |
| slaweq | any volunteer to do that? | 14:25 |
| ralonsoh | I'm on it now | 14:25 |
| slaweq | ralonsoh: thx a lot | 14:25 |
| slaweq | any other updates regarding community goals? amotoki, ralonsoh? | 14:26 |
| ralonsoh | no, thanks | 14:26 |
| amotoki | no update this week | 14:26 |
| slaweq | ok, lets move on then | 14:26 |
| slaweq | #topic Bugs | 14:26 |
| *** openstack changes topic to "Bugs (Meeting topic: networking)" | 14:26 | |
| slaweq | lucasgomes was bug deputy: http://lists.openstack.org/pipermail/openstack-discuss/2020-November/018782.html | 14:27 |
| slaweq | I went through this list today | 14:27 |
| slaweq | and I think there is couple of bugs which needs some attention | 14:27 |
| slaweq | first of all, 2 bugs related to stable branches | 14:28 |
| slaweq | https://bugs.launchpad.net/neutron/+bug/1903531 - broken upgrade - we need fix it quickly or revert patch which broke it. | 14:28 |
| openstack | Launchpad bug 1903531 in neutron "Update of neutron-server breaks compatibility to previous neutron-agent version" [Critical,Confirmed] | 14:28 |
| slaweq | we broke compatibility between agents and server with that backport :/ | 14:28 |
| slaweq | which is very bad | 14:28 |
| slaweq | and we should now either fix that in stable branches or revert patch which caused that | 14:28 |
| slaweq | any opinions which one is better? | 14:29 |
| bcafarel | yup sorry I missed that "," in rpc change :/ | 14:29 |
| ralonsoh | IMO, because we are changing the API content, we should revert the stable branch patches | 14:29 |
| slaweq | bcafarel: yes, me too :/ | 14:29 |
| bcafarel | as it is a security fix by itself additional fix would be great though | 14:29 |
| slaweq | and the worst thing is that we merged that just before Stein EM, so we can't now do new Stein release, right? | 14:30 |
| ralonsoh | I think so... | 14:31 |
| bcafarel | transition patch is not merged yet hberaud had questions (lucky us) | 14:31 |
| bcafarel | https://review.opendev.org/#/c/762404/ | 14:31 |
| slaweq | ok, so lets revert it asap and do quickly one more release of stein | 14:31 |
| slaweq | so this will be fixed in last released version | 14:32 |
| slaweq | are You ok with that? | 14:32 |
| ralonsoh | so, are we going to fix the fix or revert it for Stein? | 14:32 |
| ralonsoh | I'm not sure | 14:32 |
| slaweq | ralonsoh: I would go with revert in all stable branches | 14:32 |
| slaweq | Train, Stein, Rocky and Queens | 14:32 |
| ralonsoh | ok | 14:33 |
| slaweq | good for You? | 14:33 |
| ralonsoh | perfect | 14:33 |
| slaweq | ok, I will propose backports immediatelly | 14:33 |
| slaweq | reverts done for all stable branches | 14:34 |
| slaweq | where it was backported | 14:34 |
| slaweq | ok, good that someone found it before EM :) | 14:35 |
| ralonsoh | sure | 14:36 |
| slaweq | lets move on to the next one | 14:36 |
| slaweq | https://bugs.launchpad.net/neutron/+bug/1903689 - stadium projects and neutron-lib issue | 14:36 |
| openstack | Launchpad bug 1903689 in neutron "[stable/ussuri] Functional job fails - AttributeError: module 'neutron_lib.constants' has no attribute 'DEVICE_OWNER_DISTRIBUTED'" [Medium,New] | 14:36 |
| slaweq | bcafarel You are on it, right? | 14:36 |
| bcafarel | nice that we have the same list of bugs we wanted to talk about :) | 14:36 |
| slaweq | :) | 14:36 |
| bcafarel | slaweq: looking into it yes, not sure yet how to fix it | 14:36 |
| bcafarel | it seems neutron is missing in u-c in ussuri and newer branches | 14:37 |
| bcafarel | also, a semi-related bug here, the sibling system does not work with UT tests py38 (seen in https://review.opendev.org/#/c/743487 ) | 14:38 |
| slaweq | bcafarel: You mean here https://github.com/openstack/requirements/blob/stable/victoria/upper-constraints.txt right? | 14:38 |
| bcafarel | as that part is a bit voodoo/black magic to me, if some experts could check | 14:39 |
| bcafarel | slaweq: yes, train and before had a "neutron==xx" line | 14:39 |
| slaweq | yes, I see | 14:39 |
| slaweq | bcafarel: I see that this py38 job is failing due to this issue with neutron version | 14:42 |
| slaweq | no? | 14:42 |
| bcafarel | yes error is same but py36/36 are working (siblings working correcly?) | 14:43 |
| lajoskatona | bcafarel: it seems that this patch removed neutron from u-c file: https://review.opendev.org/553030 | 14:43 |
| bcafarel | lajoskatona looks like a good suspect! | 14:44 |
| lajoskatona | no, it is from rocky perhaps, but not sure | 14:44 |
| lajoskatona | but anyway, it seems that this happened in history :-) | 14:44 |
| slaweq | but it is later in e.g. train | 14:45 |
| slaweq | ok, bcafarel I hope You will take care of it | 14:47 |
| slaweq | if You will need any help, please ping me | 14:47 |
| bcafarel | will do! | 14:47 |
| slaweq | thx | 14:47 |
| slaweq | ok, lets move on | 14:47 |
| slaweq | next is CI related bug | 14:47 |
| slaweq | https://bugs.launchpad.net/neutron/+bug/1903985 | 14:47 |
| openstack | Launchpad bug 1903985 in neutron "[functional] Timeouts during setting link attributes in the namepaces" [High,Confirmed] | 14:47 |
| slaweq | we see it more and more IMO recently | 14:48 |
| slaweq | so we need someone who will maybe take a look at those logs and try to disable logging of some of errors in stdout/stderr | 14:48 |
| slaweq | if anyone have any cycles, that would be great | 14:49 |
| ralonsoh | this problem is specific to port.link.set_up() | 14:49 |
| ralonsoh | I think that could be a problem with the device created | 14:50 |
| slaweq | ahh, right ralonsoh | 14:50 |
| slaweq | sorry | 14:50 |
| slaweq | I was thinking about different issue | 14:50 |
| slaweq | true | 14:50 |
| slaweq | but this also happens from time to time recently :) | 14:50 |
| ralonsoh | it uses privileged.set_link_attribute | 14:50 |
| ralonsoh | sure | 14:50 |
| ralonsoh | I'll check it today | 14:50 |
| slaweq | thx | 14:50 |
| slaweq | and the last one which I wanted to mention today is | 14:51 |
| slaweq | https://bugs.launchpad.net/neutron/+bug/1904041 | 14:51 |
| openstack | Launchpad bug 1904041 in neutron "Victoria neutron-db-sync fails with I38991de2b4_source_and_destination_ip_prefix_neutron_metering_rule.py" [Undecided,New] | 14:51 |
| slaweq | which is already initially checked by ralonsoh | 14:51 |
| slaweq | thx for that | 14:51 |
| ralonsoh | I can't reproduce this issue | 14:51 |
| ralonsoh | even modifying the DB manually | 14:51 |
| slaweq | it also works fine in our CI | 14:51 |
| slaweq | on every run db migration script is run, right? | 14:51 |
| ralonsoh | yes | 14:52 |
| slaweq | ok, lets wait for some more informations and we will see | 14:52 |
| slaweq | and that are all bugs which I wanted to raise here from lucasgomes' report | 14:53 |
| slaweq | this week our bug deputy is jlibosva | 14:53 |
| slaweq | I already contacted him to remind it | 14:53 |
| slaweq | and next week is obondarev's turn | 14:53 |
| slaweq | for next week I will prepare new round | 14:53 |
| obondarev | + | 14:54 |
| * bcafarel gets ready | 14:54 | |
| slaweq | with that I'm done for today | 14:54 |
| slaweq | do You have anything else You want to discuss with the team? | 14:54 |
| ralonsoh | not from e | 14:54 |
| ralonsoh | me* | 14:54 |
| bcafarel | all good, both bugs I wanted to talk about were mentioned | 14:55 |
| njohnston | I have one thing | 14:55 |
| slaweq | njohnston: sure | 14:55 |
| njohnston | with the work getting started with the Secure RBAC popup team, do we have to think about the impact for stadium projects at this point? | 14:55 |
| slaweq | what is exactly Secure RBAC popup team? I think I missed something | 14:56 |
| njohnston | #link http://lists.openstack.org/pipermail/openstack-discuss/2020-November/018800.html | 14:57 |
| lajoskatona | it is mostly about testing, or am I wrong? | 14:57 |
| slaweq | njohnston: thx for raising this up, I will read this email as I missed it earlier | 14:58 |
| slaweq | and also I think we need amotoki to be involved in that | 14:58 |
| njohnston | It is, but for some roles that don't correspond to roles we have in traditional policies we may find some gaps or bugs | 14:58 |
| njohnston | +1 | 14:59 |
| amotoki | I just talked with gmann and am trying to catch up the discussion in the policy meeting | 14:59 |
| slaweq | amotoki: great, thx | 14:59 |
| amotoki | but we need to explore how we can cover policy testing in our repo | 15:00 |
| slaweq | so lets get back to this on our next meeting | 15:00 |
| slaweq | ok, we are out of time now | 15:01 |
| mlavalle | o/ | 15:01 |
| slaweq | thx for attending the meeting | 15:01 |
| lajoskatona | o/ | 15:01 |
| slaweq | o/ | 15:01 |
| njohnston | o/ thanks | 15:01 |
| ralonsoh | bye | 15:01 |
| amotoki | o/ | 15:01 |
| slaweq | #endmeeting | 15:01 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 15:01 | |
| openstack | Meeting ended Tue Nov 17 15:01:20 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:01 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/networking/2020/networking.2020-11-17-14.00.html | 15:01 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/networking/2020/networking.2020-11-17-14.00.txt | 15:01 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/networking/2020/networking.2020-11-17-14.00.log.html | 15:01 |
| bcafarel | o/ | 15:01 |
| slaweq | #startmeeting neutron_ci | 15:01 |
| openstack | Meeting started Tue Nov 17 15:01:38 2020 UTC and is due to finish in 60 minutes. The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:01 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:01 |
| slaweq | and welcome back :) | 15:01 |
| *** openstack changes topic to " (Meeting topic: neutron_ci)" | 15:01 | |
| openstack | The meeting name has been set to 'neutron_ci' | 15:01 |
| lajoskatona | o/ | 15:01 |
| ralonsoh | hi | 15:01 |
| bcafarel | second o/ | 15:01 |
| slaweq | lets do that one quick :) | 15:02 |
| slaweq | Grafana dashboard: http://grafana.openstack.org/dashboard/db/neutron-failure-rate | 15:02 |
| slaweq | #topic Actions from previous meetings | 15:03 |
| *** openstack changes topic to "Actions from previous meetings (Meeting topic: neutron_ci)" | 15:03 | |
| slaweq | first one | 15:03 |
| slaweq | slaweq to check failing neutron-grenade-ovn job | 15:03 |
| slaweq | I checked it | 15:03 |
| slaweq | and I found that it's failing due to the same reason why all ovn multinode jobs | 15:03 |
| slaweq | https://bugs.launchpad.net/neutron/+bug/1904117 | 15:03 |
| openstack | Launchpad bug 1904117 in neutron "Nodes in the OVN scenario multinode jobs can't talk to each other" [High,In progress] - Assigned to Slawek Kaplonski (slaweq) | 15:03 |
| slaweq | I proposed patches https://review.opendev.org/#/c/762650/ and https://review.opendev.org/#/c/762654/ | 15:03 |
| slaweq | now I'm waiting for result of the last run of it | 15:04 |
| slaweq | I hope it will work and will be accepted by zuul people | 15:04 |
| *** artom has joined #openstack-meeting-3 | 15:04 | |
| slaweq | if You have any other ideas how we can solve that problem (maybe in some easier way) than please speak up :) | 15:05 |
| slaweq | next one | 15:06 |
| slaweq | slaweq to report bug regarding error 500 in ovn functional tests | 15:06 |
| slaweq | https://bugs.launchpad.net/neutron/+bug/1903008 | 15:06 |
| openstack | Launchpad bug 1903008 in neutron "Create network failed during functional test" [High,Confirmed] | 15:06 |
| slaweq | according to ralonsoh's comment we should wait with this until we will finish migration to new engine facade | 15:07 |
| ralonsoh | right | 15:07 |
| bcafarel | which should be soon :) | 15:08 |
| slaweq | I hope so | 15:08 |
| slaweq | ok, next one | 15:08 |
| slaweq | ralonsoh to check error 500 in ovn functional tests | 15:08 |
| ralonsoh | I didn't find anything relevant sorry | 15:09 |
| ralonsoh | I don't know why is failing... | 15:09 |
| slaweq | ok, I will ask also jlibosva and lucasgomes to take a look | 15:09 |
| slaweq | ralonsoh: was there LP reported for that? | 15:10 |
| ralonsoh | no | 15:10 |
| slaweq | ok, I will report one and ping ovn folks to check it | 15:11 |
| slaweq | #action slaweq to report bug regarding errors 500 in ovn functional tests | 15:11 |
| slaweq | ok, and the last one for today | 15:12 |
| slaweq | slaweq to report LP regarding functional test timeout on set_link_attribute method | 15:12 |
| slaweq | https://bugs.launchpad.net/neutron/+bug/1903985 | 15:12 |
| openstack | Launchpad bug 1903985 in neutron "[functional] Timeouts during setting link attributes in the namepaces" [High,Confirmed] | 15:12 |
| slaweq | ok, lets move on | 15:14 |
| slaweq | #topic Stadium projects | 15:14 |
| *** openstack changes topic to "Stadium projects (Meeting topic: neutron_ci)" | 15:14 | |
| slaweq | lajoskatona: anything urgent/new regarding stadiums CI? | 15:14 |
| slaweq | ok, I guess that except this issue with capping neutron in u-c all is fine there | 15:16 |
| bcafarel | mostly yes | 15:16 |
| lajoskatona | nothing | 15:16 |
| *** lpetrut has quit IRC | 15:16 | |
| bcafarel | some open questions for stein EM transition in https://review.opendev.org/#/c/762404 but nothing important I think (I left comments for most projects) | 15:17 |
| slaweq | I will check those when this fix/revert will be ready | 15:17 |
| slaweq | ok | 15:18 |
| slaweq | so next topic | 15:18 |
| slaweq | #topic Stable branches | 15:18 |
| *** openstack changes topic to "Stable branches (Meeting topic: neutron_ci)" | 15:18 | |
| slaweq | and the same question :) | 15:18 |
| slaweq | bcafarel: anything new/urgent here? | 15:19 |
| bcafarel | everything handled already :) | 15:19 |
| slaweq | great | 15:19 |
| slaweq | so lets move on | 15:19 |
| bcafarel | I saw quite a few backports get merged last week, CI looks OK on stable branches | 15:19 |
| slaweq | good to hear that | 15:19 |
| slaweq | :) | 15:20 |
| slaweq | #topic Grafana | 15:20 |
| *** openstack changes topic to "Grafana (Meeting topic: neutron_ci)" | 15:20 | |
| slaweq | #link http://grafana.openstack.org/dashboard/db/neutron-failure-rate | 15:20 |
| slaweq | the only thing I can say from it is that numbers there aren't good in overall | 15:21 |
| slaweq | but I don't see any specific new issue there | 15:21 |
| slaweq | just that our jobs are mostly not stable still :/ | 15:21 |
| slaweq | but it's getting to be better IMO | 15:21 |
| bcafarel | with those connect/ssh fixes I think it is better in last few days? | 15:21 |
| bcafarel | at least I saw a few +2 pass by :) | 15:22 |
| slaweq | yes, and fix for checking vm's console log before ssh to instance was just merged in tempest yesterday night | 15:22 |
| slaweq | which I hope will help with many tempest jobs | 15:22 |
| slaweq | so I think we can move on to some scenario jobs' failures | 15:25 |
| slaweq | #topic Tempest/Scenario | 15:25 |
| *** openstack changes topic to "Tempest/Scenario (Meeting topic: neutron_ci)" | 15:25 | |
| slaweq | today I went through failures from last week and I found couple of issues there | 15:26 |
| slaweq | there was (or is, idk exactly) some issue with rabbitmq which didn't start properly | 15:26 |
| slaweq | but that's not on us really | 15:26 |
| slaweq | except that, failure which was seen most often was problem with hostname command, like https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_b18/753847/14/gate/neutron-tempest-plugin-scenario-linuxbridge/b18b3b6/testr_results.html | 15:27 |
| slaweq | but that should be fixed by ralonsoh | 15:27 |
| slaweq | right? | 15:27 |
| ralonsoh | I think so | 15:27 |
| bcafarel | in rechecks I think yes | 15:27 |
| ralonsoh | https://review.opendev.org/#/c/762527/ | 15:27 |
| bcafarel | do we know why hostname started returning -1 recently btw? | 15:27 |
| slaweq | bcafarel: I think it was like that before from time to time | 15:28 |
| slaweq | and recently this test was marked as unstable due to other issues | 15:28 |
| bcafarel | ok I prefer that :) | 15:28 |
| slaweq | so we probably didn't saw it really | 15:28 |
| *** obondarev has quit IRC | 15:29 | |
| ralonsoh | we can skip that reading the hostname file directly | 15:29 |
| ralonsoh | that's not perfect, but works for us | 15:29 |
| slaweq | if that works I'm ok with it | 15:29 |
| slaweq | it's just a test :) | 15:29 |
| slaweq | ok, other issue which I saw is timeout waiting for instance | 15:31 |
| slaweq | like: https://10c0cf0314a8bd66c0e4-c578cacb39dd1edf606b634ec77d1998.ssl.cf5.rackcdn.com/762654/5/check/neutron-tempest-plugin-scenario-openvswitch-iptables_hybrid/78884bd/testr_results.html | 15:31 |
| slaweq | I think lajoskatona also saw it recently | 15:31 |
| slaweq | and IIRC it's always in the same test, which is strange for me | 15:31 |
| lajoskatona | yeah, once or twice this week perhaps | 15:32 |
| slaweq | lajoskatona: did You check with Nova what can be wrong there? | 15:32 |
| ralonsoh | this could be because of the advance image | 15:32 |
| lajoskatona | I asked gibi to help me out ith n-cpu logs and He said that one slow thing was the conversion from qcow2 to raw format, at least from log | 15:32 |
| lajoskatona | yeah I hade the same feeling | 15:32 |
| slaweq | ok | 15:33 |
| lajoskatona | if you check on canonical page it is slightly bigger than previous images | 15:33 |
| slaweq | so we can try to increase this build timeout (but how much would be enough) | 15:33 |
| ralonsoh | if this is because of the conversion, can we point to the raw format image? | 15:33 |
| slaweq | or try to look for some smaller image maybe | 15:33 |
| slaweq | ralonsoh: but raw will be much bigger, no? | 15:34 |
| ralonsoh | maybe... | 15:34 |
| lajoskatona | yes, thats bigger I suppose, but no conversion | 15:34 |
| ralonsoh | but I don't see it in the repos | 15:34 |
| ralonsoh | https://cloud-images.ubuntu.com/bionic/current/ | 15:35 |
| slaweq | maybe this one https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64-root.tar.xz ? | 15:35 |
| slaweq | idk exactly what are all those images really :) | 15:36 |
| ralonsoh | ok so the problems are the conversion and the resources (more ram and disk) | 15:36 |
| ralonsoh | do you know if we support something else apart from ubuntu and cirros? | 15:37 |
| slaweq | we should have a bit more resources after https://review.opendev.org/#/c/762582/ and https://review.opendev.org/#/c/762539/ will be merged | 15:37 |
| ralonsoh | we can also try to use less workers | 15:38 |
| slaweq | but we need also https://review.opendev.org/#/c/762622/ to merge all of that | 15:38 |
| ralonsoh | that will take more time but with less probability of failure | 15:38 |
| slaweq | ralonsoh: test workers or neutron workers? | 15:38 |
| ralonsoh | test | 15:38 |
| ralonsoh | ncpu - 1, for example | 15:39 |
| slaweq | we can try but I think we already use only 2 | 15:39 |
| slaweq | IIRC | 15:39 |
| ralonsoh | really? | 15:39 |
| ralonsoh | ok | 15:39 |
| slaweq | sorry, 4 | 15:40 |
| slaweq | so we can try only 2 | 15:40 |
| ralonsoh | or 3 yes | 15:40 |
| ralonsoh | that will take more time but less problems, I think so | 15:40 |
| slaweq | especially in neutron-tempest-plugin jobs where there is no so many tests to run | 15:40 |
| slaweq | ralonsoh: will You propose a patch or do You want me to do it? | 15:40 |
| ralonsoh | sure | 15:40 |
| ralonsoh | I'll do it | 15:40 |
| slaweq | thx | 15:40 |
| slaweq | #action ralonsoh will decrease number of test workers in scenario jobs | 15:41 |
| slaweq | I also found other interesting failure with server | 15:41 |
| slaweq | but only once so far | 15:41 |
| slaweq | https://6ad68def19a9c3e3c7f7-a757501b1a7ef7a48e849fadd8ea0086.ssl.cf2.rackcdn.com/759657/1/check/neutron-tempest-plugin-scenario-ovn/3ce85d5/testr_results.html | 15:41 |
| slaweq | it seems it was timeout during server termination | 15:42 |
| slaweq | did You saw something like that before? | 15:42 |
| ralonsoh | only testing manually | 15:43 |
| slaweq | ok, lets see how it will be and if it will repeat more often, we will report bug for that | 15:43 |
| slaweq | and we will see :) | 15:43 |
| slaweq | and that's all what I have for today | 15:44 |
| slaweq | periodic jobs are ok recently | 15:44 |
| slaweq | do You have anything else You want to talk today? | 15:44 |
| ralonsoh | no | 15:45 |
| slaweq | if not, I will give You 15 minutes back | 15:45 |
| bcafarel | nice to be able to wrap the meeting in less time :) | 15:45 |
| slaweq | ok, thx for attending | 15:45 |
| ralonsoh | bye! | 15:46 |
| slaweq | have a great evening and see You online tomorrow :) | 15:46 |
| slaweq | o/ | 15:46 |
| slaweq | #endmeeting | 15:46 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 15:46 | |
| openstack | Meeting ended Tue Nov 17 15:46:09 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:46 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-11-17-15.01.html | 15:46 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-11-17-15.01.txt | 15:46 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-11-17-15.01.log.html | 15:46 |
| lajoskatona | o/ | 15:47 |
| *** lajoskatona has left #openstack-meeting-3 | 15:49 | |
| *** artom has quit IRC | 15:54 | |
| *** bnemec has quit IRC | 16:16 | |
| *** artom has joined #openstack-meeting-3 | 16:28 | |
| *** bnemec has joined #openstack-meeting-3 | 16:50 | |
| *** mlavalle has quit IRC | 17:18 | |
| *** e0ne has quit IRC | 17:22 | |
| *** mlavalle has joined #openstack-meeting-3 | 17:31 | |
| *** ralonsoh has quit IRC | 22:03 | |
| *** yamamoto has quit IRC | 23:14 | |
| *** yamamoto has joined #openstack-meeting-3 | 23:19 | |
| *** slaweq has quit IRC | 23:21 | |
| *** tosky has quit IRC | 23:58 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!