| *** zzzeek has joined #openstack-infra | 00:01 | |
| *** rlandy has quit IRC | 00:21 | |
| *** armax has quit IRC | 00:33 | |
| *** zzzeek has quit IRC | 00:38 | |
| *** zzzeek has joined #openstack-infra | 00:42 | |
| *** gyee has quit IRC | 00:50 | |
| *** d34dh0r53 has quit IRC | 00:57 | |
| *** armax has joined #openstack-infra | 01:21 | |
| *** eandersson has quit IRC | 01:44 | |
| *** eandersson has joined #openstack-infra | 01:45 | |
| *** vishalmanchanda has joined #openstack-infra | 01:58 | |
| *** zzzeek has quit IRC | 02:14 | |
| *** stevebaker_ is now known as stevebaker | 02:15 | |
| *** zzzeek has joined #openstack-infra | 02:15 | |
| *** artom has quit IRC | 02:28 | |
| *** zzzeek has quit IRC | 02:45 | |
| *** zzzeek has joined #openstack-infra | 02:48 | |
| *** rcernin has quit IRC | 02:50 | |
| *** ramishra has joined #openstack-infra | 02:59 | |
| *** rcernin has joined #openstack-infra | 02:59 | |
| *** armax has quit IRC | 03:02 | |
| *** stevebaker has quit IRC | 03:19 | |
| *** stevebaker has joined #openstack-infra | 03:19 | |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Add Fedora 32 builds https://review.opendev.org/750642 | 03:45 |
|---|---|---|
| openstackgerrit | Merged openstack/project-config master: Add Fedora 32 builds https://review.opendev.org/750642 | 04:06 |
| *** dave-mccowan has quit IRC | 04:12 | |
| *** ykarel|away has joined #openstack-infra | 04:22 | |
| *** ykarel|away is now known as ykarel | 04:23 | |
| *** ykarel has quit IRC | 04:30 | |
| *** bhagyashris|rove is now known as bhagyashri|rover | 04:32 | |
| *** evrardjp has quit IRC | 04:33 | |
| *** evrardjp has joined #openstack-infra | 04:33 | |
| *** ykarel has joined #openstack-infra | 04:37 | |
| *** ykarel_ has joined #openstack-infra | 04:51 | |
| *** ykarel has quit IRC | 04:54 | |
| *** matt_kosut has joined #openstack-infra | 05:01 | |
| *** sorrison has joined #openstack-infra | 05:03 | |
| sorrison | Hi, I'm converting a project (networking-midonet) over to zuulv3 and I'm having some issues with the multinode. Wondering if I can get SSH access to the nodes to debug further? | 05:04 |
| sorrison | jhesketh you around? | 05:08 |
| jhesketh | sorrison: yes, but I'm not longer infra-root/core sorry | 05:09 |
| sorrison | ah ok, hi by the way | 05:10 |
| sorrison | know anyone in this timezone who is? | 05:10 |
| jhesketh | ianw might be around | 05:12 |
| jhesketh | also hi! :-) | 05:12 |
| *** ykarel__ has joined #openstack-infra | 05:12 | |
| *** ykarel_ has quit IRC | 05:14 | |
| *** jtomasek has joined #openstack-infra | 05:18 | |
| *** ysandeep|away is now known as ysandeep | 05:18 | |
| *** ykarel__ is now known as ykarel | 05:22 | |
| *** jtomasek has quit IRC | 05:22 | |
| *** lmiccini has joined #openstack-infra | 05:42 | |
| ianw | sorrison: what's the job / change? | 05:54 |
| sorrison | https://review.opendev.org/#/c/749857/ | 05:55 |
| sorrison | its the networking-midonet-ml2-tempest-multinode | 05:55 |
| sorrison | my theory is that the multinode bridge playbook is somehow conflicting with what midonet is doing but I haven't been able to replicate locally | 05:56 |
| *** zzzeek has quit IRC | 05:57 | |
| ianw | one sec, let me log in and put a hold on it | 06:00 |
| *** zzzeek has joined #openstack-infra | 06:01 | |
| *** ykarel_ has joined #openstack-infra | 06:01 | |
| *** ykarel has quit IRC | 06:03 | |
| sorrison | It's not running, want me to recheck? | 06:05 |
| ianw | sorrison: this you? -> https://github.com/sorrison.keys | 06:05 |
| sorrison | yip thats me | 06:05 |
| ianw | ok, it might take a little. the 0600 periodic job kick offs are always fun | 06:09 |
| *** ricolin has quit IRC | 06:10 | |
| sorrison | could always pick this up tomorrow morning sometime instead | 06:15 |
| *** ykarel_ is now known as ykarel | 06:15 | |
| ianw | sorrison: it should start in a sec and then the nodes will hold till you shut them down or we kill them | 06:18 |
| sorrison | ok great, thanks | 06:20 |
| sorrison | ianw: what username do I log in as? | 06:24 |
| ianw | sorrison: try roo@198.72.124.18 | 06:26 |
| ianw | root even | 06:26 |
| sorrison | I'm in | 06:27 |
| *** eolivare has joined #openstack-infra | 06:28 | |
| *** ralonsoh has joined #openstack-infra | 06:34 | |
| *** dklyle has quit IRC | 06:43 | |
| *** slaweq has joined #openstack-infra | 06:44 | |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: Remove legacy-tempest-dsvm-zeromq-multibackend https://review.opendev.org/750878 | 06:45 |
| *** hashar has joined #openstack-infra | 06:50 | |
| *** jaosorior has joined #openstack-infra | 06:58 | |
| openstackgerrit | Merged openstack/project-config master: pynotedb: remove project from infrastructure systems https://review.opendev.org/750635 | 06:58 |
| *** jtomasek has joined #openstack-infra | 06:59 | |
| sorrison | ianw: Could I get access to 198.72.124.208 too please (the other node in the same job) | 07:07 |
| ianw | sorrison: umm, you should be able to bounce via the first node | 07:11 |
| sorrison | Ah, I see I can get to the other one through the internal network | 07:11 |
| sorrison | :-) | 07:11 |
| *** priteau has joined #openstack-infra | 07:16 | |
| *** tosky has joined #openstack-infra | 07:32 | |
| AJaeger | config-core, please review https://review.opendev.org/750878 to remove a broken job (repo was retired) | 07:32 |
| *** jpena|off is now known as jpena | 07:40 | |
| *** jcapitao has joined #openstack-infra | 07:42 | |
| *** priteau has quit IRC | 07:47 | |
| *** andrewbonney has joined #openstack-infra | 07:47 | |
| *** zzzeek has quit IRC | 07:48 | |
| *** zzzeek has joined #openstack-infra | 07:50 | |
| *** priteau has joined #openstack-infra | 07:53 | |
| frickler | lyarwood: ianw: the fedora32 image seem to have built successfully. does it also need some new repo mirror setup? | 07:55 |
| *** xek has joined #openstack-infra | 07:55 | |
| *** xek has quit IRC | 08:09 | |
| *** xek has joined #openstack-infra | 08:10 | |
| *** lucasagomes has joined #openstack-infra | 08:11 | |
| *** xek has quit IRC | 08:17 | |
| *** jcapitao has quit IRC | 08:27 | |
| *** jaicaa has quit IRC | 08:29 | |
| *** jaicaa has joined #openstack-infra | 08:31 | |
| *** ysandeep is now known as ysandeep|lunch | 08:34 | |
| *** ykarel_ has joined #openstack-infra | 08:35 | |
| *** rcernin has quit IRC | 08:35 | |
| *** ykarel has quit IRC | 08:36 | |
| *** jcapitao has joined #openstack-infra | 08:36 | |
| *** ykarel_ is now known as ykarel|lunch | 08:40 | |
| *** jcapitao has quit IRC | 08:41 | |
| *** jcapitao has joined #openstack-infra | 08:42 | |
| *** jtomasek has quit IRC | 08:45 | |
| *** slaweq has quit IRC | 08:46 | |
| *** gfidente has joined #openstack-infra | 08:49 | |
| *** derekh has joined #openstack-infra | 08:52 | |
| *** ralonsoh has quit IRC | 08:54 | |
| *** ralonsoh has joined #openstack-infra | 08:54 | |
| *** ricolin has joined #openstack-infra | 09:09 | |
| *** ykarel|lunch has quit IRC | 09:12 | |
| *** dtantsur|afk is now known as dtantsur | 09:14 | |
| *** ykarel has joined #openstack-infra | 09:19 | |
| lyarwood | frickler: I'm not sure if we've done that before with Fedora, any idea where that would be configured and I can take a look? | 09:23 |
| frickler | lyarwood: I was just looking for that ;) https://review.opendev.org/712680 seems to be the reference plus some followup for the updates/ part | 09:27 |
| lyarwood | I think that should be *remove* releases/29 FWIW | 09:29 |
| lyarwood | frickler: do you already have a change in progress? | 09:29 |
| lyarwood | https://review.opendev.org/#/c/713177/ already removed f29 | 09:30 |
| lyarwood | argh configure-swap is failing still on f32 | 09:32 |
| lyarwood | it started to fail on f31 after a recent kernel update | 09:32 |
| lyarwood | appears fallocate on ext4 now leaves holes in the filesystem that borks swapon | 09:33 |
| * lyarwood pushes a fix | 09:33 | |
| *** ysandeep|lunch is now known as ysandeep | 09:35 | |
| frickler | lyarwood: https://review.opendev.org/713169 is the fup I was talking about. removing f30 will have to wait until no one uses it anymore | 09:37 |
| frickler | lyarwood: also the swap failure seem to happen only on a subset of our cloud providers, but if you can fix it, that's great. I did look into it some time ago but didn't follow up | 09:38 |
| lyarwood | frickler: https://bugzilla.redhat.com/show_bug.cgi?id=1827115 - it doesn't cover what has actually changed just that it's borked for other people, I can reproduce locally as well | 09:39 |
| openstack | bugzilla.redhat.com bug 1827115 in kernel "[upstream kernel]swapon failed if the swap file is created by fallocate in ext4 partition" [Medium,New] - Assigned to kernel-maint | 09:39 |
| lyarwood | the workaround doesn't appear to work for me however so meh | 09:39 |
| frickler | lyarwood: ah, now I seem to remember that we discussed swapping swapfile generation to dd and also reduce the size to 1G in order to spend less time on it | 09:43 |
| lyarwood | frickler: kk, I can push a change for this now as I can't find a workaround for this locally tbh | 09:52 |
| * lyarwood assumes all distros will hit this eventually | 09:52 | |
| openstackgerrit | Lee Yarwood proposed openstack/openstack-zuul-jobs master: configure-swap: Always use dd and limit swap size to 1024MB https://review.opendev.org/750941 | 09:55 |
| *** hashar has quit IRC | 09:58 | |
| *** hashar has joined #openstack-infra | 09:58 | |
| frickler | lyarwood: there's a trailing space in that, otherwise I agree | 09:59 |
| lyarwood | gah sorry | 10:01 |
| openstackgerrit | Lee Yarwood proposed openstack/openstack-zuul-jobs master: configure-swap: Always use dd and limit swap size to 1024MB https://review.opendev.org/750941 | 10:02 |
| *** slaweq has joined #openstack-infra | 10:20 | |
| *** StevenK has quit IRC | 10:21 | |
| *** jaosorior has quit IRC | 10:24 | |
| *** Lucas_Gray has joined #openstack-infra | 10:32 | |
| *** rcernin has joined #openstack-infra | 10:39 | |
| *** dmellado has quit IRC | 11:18 | |
| *** StevenK_ has joined #openstack-infra | 11:19 | |
| *** dmellado has joined #openstack-infra | 11:19 | |
| sean-k-mooney | does disk image builder use anything for tracking features by the way. in my copious free time.... i have started playing with creating a alpine element and i need to create a apk element next for the package installer | 11:20 |
| *** jcapitao is now known as jcapitao_lunch | 11:21 | |
| sean-k-mooney | the end goal is to build a replacment for cirros | 11:22 |
| sean-k-mooney | given its more or less un maintained now | 11:22 |
| *** StevenK_ is now known as StevenK | 11:23 | |
| *** artom has joined #openstack-infra | 11:31 | |
| *** jpena is now known as jpena|lunch | 11:31 | |
| *** dmellado has quit IRC | 11:33 | |
| *** dmellado has joined #openstack-infra | 11:34 | |
| *** thiagop has joined #openstack-infra | 11:42 | |
| *** dave-mccowan has joined #openstack-infra | 11:46 | |
| *** arxcruz|ruck is now known as arxcruz|pto | 12:05 | |
| *** rlandy has joined #openstack-infra | 12:06 | |
| *** dmellado has quit IRC | 12:10 | |
| *** rfolco has joined #openstack-infra | 12:11 | |
| *** dmellado has joined #openstack-infra | 12:11 | |
| *** jcapitao_lunch is now known as jcapitao | 12:16 | |
| *** rfolco is now known as rfolco|ruck | 12:18 | |
| *** hashar has quit IRC | 12:31 | |
| sean-k-mooney | smcginnis: frickler ^ any comment on diskimage-build "feature" tracking by the way | 12:34 |
| sean-k-mooney | i was just going to submit the patches when it worked and figure it out then | 12:34 |
| *** hashar has joined #openstack-infra | 12:37 | |
| *** stephenfin has joined #openstack-infra | 12:37 | |
| *** jpena|lunch is now known as jpena | 12:39 | |
| *** hashar has quit IRC | 12:40 | |
| *** ykarel_ has joined #openstack-infra | 12:48 | |
| *** ykarel has quit IRC | 12:51 | |
| sean-k-mooney | oh my vpn is down | 12:54 |
| sean-k-mooney | i tought downstream irc was terrible quite today | 12:55 |
| *** derekh has quit IRC | 12:58 | |
| fungi | sean-k-mooney: probably could get by with just good commit messages and talking things through in irc, diskimage-builder has an #openstack-dib irc channel of their own and uses https://launchpad.net/diskimage-builder for defect tracking so might also use that to do feature planning | 13:00 |
| fungi | also mailing lists are a fine place to do feature planning, in my opinion | 13:00 |
| sean-k-mooney | cool | 13:01 |
| sean-k-mooney | i have the basic downloading of the mini root filesystem working | 13:01 |
| sean-k-mooney | all of the other image they provide are ISOs for some reason | 13:02 |
| sean-k-mooney | so i was going to use the filesystem tar as a base eventhough that is normally for containers | 13:02 |
| sean-k-mooney | but sicne we are building via a chroot i think it makes sense | 13:02 |
| sean-k-mooney | what im currenlty failing on is the lack of install-pacakges but that just because i have not create the apk element yet | 13:03 |
| sean-k-mooney | anyway ill pop into the dib channel and say hi at some point | 13:03 |
| sean-k-mooney | the only disadvantage of the Mini root filesystem is i need to remember to add a kernel and bootloader | 13:05 |
| *** priteau has quit IRC | 13:11 | |
| *** sorrison has quit IRC | 13:14 | |
| *** sorrison has joined #openstack-infra | 13:14 | |
| *** priteau has joined #openstack-infra | 13:16 | |
| *** xek has joined #openstack-infra | 13:18 | |
| *** lmiccini has quit IRC | 13:21 | |
| *** lmiccini has joined #openstack-infra | 13:23 | |
| openstackgerrit | Sean McGinnis proposed openstack/project-config master: Remove noop jobs for python-adjutantclient https://review.opendev.org/750990 | 13:25 |
| *** ykarel_ is now known as ykarel | 13:25 | |
| *** Goneri has joined #openstack-infra | 13:26 | |
| *** piotrowskim has joined #openstack-infra | 13:28 | |
| *** armax has joined #openstack-infra | 13:38 | |
| *** hashar has joined #openstack-infra | 13:48 | |
| *** d34dh0r53 has joined #openstack-infra | 14:01 | |
| *** thiagop has quit IRC | 14:15 | |
| AJaeger | fungi, clarkb, could you check the mailthread at http://lists.openstack.org/pipermail/openstack-discuss/2020-September/017169.html about cinder old branches, please? What's the best way forward here with the broken jobs? | 14:17 |
| *** ykarel is now known as ykarel|away | 14:18 | |
| *** jtomasek has joined #openstack-infra | 14:23 | |
| AJaeger | config-core, please review https://review.opendev.org/750878 to remove one broken job | 14:25 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Retire devstack-plugin-pika project https://review.opendev.org/748712 | 14:40 |
| frickler | config-core, please also review https://review.opendev.org/750941 to the the issues with swap setup that we've been seeing on fedora for a while | 14:40 |
| frickler | s/the the/fix the/ | 14:41 |
| AJaeger | config-core, and review 748712 to retire another repo, please | 14:41 |
| *** ykarel|away has quit IRC | 14:45 | |
| *** d34dh0r53 has quit IRC | 14:45 | |
| *** d34dh0r53 has joined #openstack-infra | 14:46 | |
| fungi | AJaeger: i was following it, sounds like there's some concern that deleting those branches will have a cascade effect for devstack jobs on similar branches of other integrated openstack services? | 14:50 |
| *** dklyle has joined #openstack-infra | 14:54 | |
| *** sreejithp has joined #openstack-infra | 14:56 | |
| AJaeger | fungi: yeah, that's the concern. I wonder what to do with the broken jobs, I would like to get that solved soonish. | 14:58 |
| clarkb | deleting branches that define jobs causes zuul to s top running thosejobs in other projects iirc | 15:00 |
| clarkb | so for example grenade can delete its jobs on the oldeat branch and that oldest branch across openstack will stop running the grenade jobs | 15:00 |
| fungi | sure, more concerned about deleting cinder branches which devstack is trying to checkout for testing nova changes on related branches there | 15:01 |
| fungi | i suppose devstack needs to switch to checking out the eol tags for those first | 15:02 |
| clarkb | or disable cinder | 15:04 |
| *** lmiccini has quit IRC | 15:08 | |
| AJaeger | so, do we care about those broken jobs that I try to remove - or wait until those branches are really deleted? | 15:11 |
| AJaeger | I wanted to have again at some point zero Zuul failures in the openstack tenant | 15:11 |
| *** __ministry1 has joined #openstack-infra | 15:15 | |
| *** __ministry1 has quit IRC | 15:16 | |
| *** hashar has quit IRC | 15:17 | |
| *** hongbin has joined #openstack-infra | 15:25 | |
| clarkb | If openstack wants to delete the branches together we should land the fixes in cinder | 15:32 |
| AJaeger | clarkb: could you followup on the thread then, please? | 15:42 |
| clarkb | ya I am doing a start of school year thing in a minute then should be back to help in an hour or so | 15:43 |
| AJaeger | clarkb: thanks, this is not urgent - just wanted to discuss it while it's on my mind ;) | 15:45 |
| *** rcernin has quit IRC | 15:50 | |
| *** Lucas_Gray has quit IRC | 15:52 | |
| *** lucasagomes has quit IRC | 16:02 | |
| *** jcapitao has quit IRC | 16:16 | |
| *** dtantsur is now known as dtantsur|afk | 16:22 | |
| *** ysandeep is now known as ysandeep|away | 16:25 | |
| openstackgerrit | Oleksandr Kozachenko proposed openstack/project-config master: Add some more monasca projects in vexxhost tenant https://review.opendev.org/751046 | 16:31 |
| *** hamalq has joined #openstack-infra | 16:32 | |
| openstackgerrit | Oleksandr Kozachenko proposed openstack/project-config master: Add some more monasca projects in vexxhost tenant https://review.opendev.org/751046 | 16:32 |
| *** dmellado_ has joined #openstack-infra | 16:33 | |
| *** dmellado has quit IRC | 16:34 | |
| *** dmellado_ is now known as dmellado | 16:34 | |
| *** artom has quit IRC | 16:34 | |
| *** dmellado has quit IRC | 16:39 | |
| *** dmellado has joined #openstack-infra | 16:39 | |
| clarkb | AJaeger: [cinder][stable] branch freeze for ocata, pike <- that thread right? | 16:41 |
| *** dmellado has quit IRC | 16:49 | |
| *** jpena is now known as jpena|off | 16:49 | |
| *** dmellado has joined #openstack-infra | 16:55 | |
| elod | clarkb: does force-merge works even if the branch has *-eol tag? ( regarding http://lists.openstack.org/pipermail/openstack-discuss/2020-September/017183.html ) | 16:58 |
| elod | if yes, then I'll remove my -2 and let Cinder team decide | 16:58 |
| clarkb | elod: if the branch is still open (it must be if zuul si complaining) then yes | 16:58 |
| clarkb | but why did they eol tag it and not delete the branch? | 16:58 |
| clarkb | sounds like that is the actual issue. Those two things should happen together | 16:58 |
| elod | yes, that was my understanding too | 16:59 |
| clarkb | previously to the extended maintenance changes we'd usually give someone like tonyb powers to do them together | 16:59 |
| clarkb | (via gerrit acl changes) | 16:59 |
| elod | I guess Cinder thought other projects have already eol'd their ocata branches | 16:59 |
| clarkb | if they've already tagged it maybe the best thing is to delete the branchse and openstack deals with the fallout? | 17:00 |
| clarkb | fungi: ^ thoughts? | 17:01 |
| elod | and realized that they'd be the first 'core' project to EOL, which could cause issues | 17:01 |
| fungi | yes, the idea behind the -eol tag was that no new changes would be merged to that branch (most easily enforced by deleting the branch itself) | 17:02 |
| elod | clarkb: Brian wrote it'd be good to wait with the delete after RC >>> http://lists.openstack.org/pipermail/openstack-discuss/2020-September/017171.html | 17:02 |
| frickler | clarkb: I agree with deleting the branches, if other projects still want to test eol branches, it is up to them to amend their jobs | 17:02 |
| fungi | if we merge changes after the -eol tag and then delete the branch, those changes will be orphaned (no longer appear in the history of any branch or tag) | 17:02 |
| *** gyee has joined #openstack-infra | 17:03 | |
| frickler | well we could tag -eol2 but that might be even more confusing | 17:04 |
| fungi | it's so end-of-lifed we tagged it twice! | 17:04 |
| fungi | i wonder if rosmaita's concern for disrupting development this close to release is really warranted, seeing as how it would at worst disrupt development for other stable branches not the branch which is being groomed for release | 17:04 |
| elod | my question: if those experimental job removal patches are not merged for a couple of weeks more, does that cause any issues anywhere? | 17:05 |
| clarkb | elod: it causes zuul to continue to complain | 17:05 |
| clarkb | its not critical and waiting for the RC is probably sane. I think if we take anything from this its that branches must be deleted when eol tagged | 17:06 |
| clarkb | this not happening is the underlying bug | 17:06 |
| elod | clarkb: I agree | 17:06 |
| fungi | also one of the reasons we took to deleting eol branches long ago was so that people pulling in their local clones would actually get an error when the remote branch ceased to exist, and that would hopefully get their attention to notice the -eol tag | 17:07 |
| *** gfidente is now known as gfidente|afk | 17:07 | |
| elod | fungi: that sounds logical | 17:09 |
| clarkb | also we might consider deleting zuul configs as a pre eol tag step | 17:10 |
| elod | clarkb: yes, I was thinking about that in the past | 17:10 |
| *** dmellado has quit IRC | 17:12 | |
| elod | clarkb: but during some discussions we agreed the the process is enough if just contain the step to remove the job definitions *outside* of the "to be EOL'd" branch, which are not used anymore | 17:12 |
| openstackgerrit | Merged openstack/project-config master: Add some more monasca projects in vexxhost tenant https://review.opendev.org/751046 | 17:12 |
| *** dmellado has joined #openstack-infra | 17:13 | |
| clarkb | ya if we're deleting the branch it shouldn't be necessary. Would mostly be a safety net for when we don't delete the branch | 17:13 |
| elod | clarkb: i can add that to the process steps (between 2 and 3) if you agree | 17:13 |
| clarkb | probably just focus on the branch deletion and that is more important | 17:14 |
| elod | clarkb: here I mean: https://docs.openstack.org/project-team-guide/stable-branches.html#end-of-life | 17:14 |
| clarkb | ya that alreadys says the branch should be deleted so we should be ok | 17:15 |
| clarkb | just need to follow the process | 17:15 |
| *** tonyb has quit IRC | 17:15 | |
| elod | clarkb: ack | 17:15 |
| elod | clarkb, fungi: so I guess we can wait for the RC and then delete cinder's stable/ocata and stable/pike | 17:16 |
| fungi | that works for me | 17:20 |
| elod | ok, thanks for the clarification, I'll answer to clarkb's mail on the mailing list that let's wait for the RC | 17:23 |
| elod | just to have it there, too :) | 17:24 |
| *** xek has quit IRC | 17:25 | |
| *** dmellado has quit IRC | 17:27 | |
| openstackgerrit | Clark Boylan proposed openstack/project-config master: Pause image builds on nb04.opendev.org https://review.opendev.org/751066 | 17:31 |
| *** Lucas_Gray has joined #openstack-infra | 17:42 | |
| *** dmellado has joined #openstack-infra | 17:46 | |
| *** tonyb has joined #openstack-infra | 17:57 | |
| *** thiagop has joined #openstack-infra | 17:59 | |
| *** thiagop has quit IRC | 18:04 | |
| *** thiagop has joined #openstack-infra | 18:05 | |
| *** artom has joined #openstack-infra | 18:09 | |
| *** gmann is now known as gmann_afk | 18:11 | |
| *** andrewbonney has quit IRC | 18:13 | |
| openstackgerrit | Merged openstack/project-config master: Pause image builds on nb04.opendev.org https://review.opendev.org/751066 | 18:14 |
| smcginnis | fungi: Wow, I had never noticed the Issues and Proposed changes links on opendev.org repos. Was that always there? | 18:31 |
| clarkb | smcginnis: they've been there for a while | 18:31 |
| fungi | smcginnis: maybe 6 months or so | 18:31 |
| smcginnis | Huh | 18:31 |
| clarkb | but weren't there when we first rolled out gitea | 18:31 |
| smcginnis | I like it. | 18:32 |
| fungi | thanks! i don't think i implemented either of those things, but just nice to see a service i help run get some compliments occasionally | 18:34 |
| *** artom has quit IRC | 18:43 | |
| clarkb | I think ttx did that work (at least for the changes) | 18:44 |
| fungi | that sounds right | 18:48 |
| *** rfolco has joined #openstack-infra | 18:54 | |
| *** piotrowskim has quit IRC | 18:54 | |
| *** smarcet has joined #openstack-infra | 18:56 | |
| *** rfolco|ruck has quit IRC | 18:56 | |
| *** hashar has joined #openstack-infra | 19:01 | |
| *** xek has joined #openstack-infra | 19:07 | |
| AJaeger | clarkb: yes, that thread | 19:10 |
| *** rfolco is now known as rfolco|ruck | 19:12 | |
| *** priteau has quit IRC | 19:16 | |
| AJaeger | config-core, please review https://review.opendev.org/750941 https://review.opendev.org/748712 https://review.opendev.org/749834 https://review.opendev.org/750643 https://review.opendev.org/750878 | 19:16 |
| *** hamalq_ has joined #openstack-infra | 19:23 | |
| *** hamalq has quit IRC | 19:24 | |
| *** ralonsoh has quit IRC | 19:27 | |
| *** slaweq has quit IRC | 19:53 | |
| clarkb | I've approved https://review.opendev.org/#/c/750941/2 and we should try to remembre that if job runtimes get worse that may be one to revisit | 19:53 |
| clarkb | (it converts swap file creation from fallocate to dd) | 19:54 |
| openstackgerrit | Merged openstack/openstack-zuul-jobs master: configure-swap: Always use dd and limit swap size to 1024MB https://review.opendev.org/750941 | 20:05 |
| openstackgerrit | Merged openstack/project-config master: Retire devstack-plugin-pika project https://review.opendev.org/748712 | 20:05 |
| AJaeger | thanks, clarkb | 20:05 |
| *** matt_kosut has quit IRC | 20:06 | |
| *** xek has quit IRC | 20:08 | |
| openstackgerrit | Merged openstack/openstack-zuul-jobs master: Remove legacy-tempest-dsvm-zeromq-multibackend https://review.opendev.org/750878 | 20:10 |
| openstackgerrit | Merged openstack/project-config master: Add openstack/osops https://review.opendev.org/749834 | 20:18 |
| openstackgerrit | Merged openstack/project-config master: Set neutron-lib stable ACLs https://review.opendev.org/750643 | 20:18 |
| *** hashar has quit IRC | 20:33 | |
| *** openstackgerrit has quit IRC | 20:36 | |
| *** dklyle has quit IRC | 20:40 | |
| *** david-lyle has joined #openstack-infra | 20:40 | |
| *** artom has joined #openstack-infra | 20:46 | |
| *** thiagop has quit IRC | 20:55 | |
| *** outbrito has joined #openstack-infra | 20:55 | |
| *** david-lyle is now known as dklyle | 20:56 | |
| *** rfolco|ruck has quit IRC | 21:00 | |
| *** jtomasek has quit IRC | 21:11 | |
| *** outbrito has quit IRC | 21:11 | |
| *** openstackgerrit has joined #openstack-infra | 21:20 | |
| openstackgerrit | Sean McGinnis proposed openstack/openstack-zuul-jobs master: Run openstack-tox-lower-constraints on bionic https://review.opendev.org/751107 | 21:20 |
| *** gfidente|afk has quit IRC | 21:21 | |
| *** rosmaita has joined #openstack-infra | 21:37 | |
| rosmaita | hello infra people | 21:39 |
| clarkb | hello | 21:39 |
| rosmaita | can i get some eyes on: https://review.opendev.org/#/c/751107/ ? | 21:39 |
| rosmaita | this is why: | 21:39 |
| rosmaita | https://zuul.opendev.org/t/openstack/builds?job_name=openstack-tox-lower-constraints&project=openstack%2Fcinder&branch=master | 21:39 |
| clarkb | thta makes sense | 21:40 |
| rosmaita | yes, and we are seriously stuck right at feature freeze time | 21:41 |
| rosmaita | gmann has been trying to sort out what the problem is, our py38 job runs fine in focal, but when the lower-constraints job runs those same tests, no dice | 21:42 |
| smcginnis | There are a bunch of other repos failing now because of running on focal too. | 21:42 |
| clarkb | its because you specifically run on python3.6 in the job but python3.6 doesn't exist on focal | 21:43 |
| clarkb | ERROR: InterpreterNotFound: python3.6 | 21:43 |
| rosmaita | that was from an experiment | 21:43 |
| smcginnis | Problem was running under py38. | 21:43 |
| rosmaita | we just use basepython=python3 normally | 21:43 |
| clarkb | is there a specific failure that is representative then? | 21:43 |
| clarkb | it could be that the constraints set a version that explicitly doesn't work with 3.8 | 21:44 |
| clarkb | usually we have version specific constraints to deal with that | 21:44 |
| rosmaita | https://review.opendev.org/#/c/743080/ | 21:44 |
| clarkb | because constraints for 3.6 may be different than 3.8 | 21:44 |
| smcginnis | Well, part of this too is lower-constraints should be testing our minimum versions, so it kind of doesn't make sense to run on the highest version runtime we have. | 21:45 |
| clarkb | andopening logs hasused all the memory again. I really need a new laptop | 21:45 |
| rosmaita | i think sean's patch makes sense in general | 21:46 |
| rosmaita | but at least it can get cinder un-hosed for a while, while we figure out what to do next | 21:47 |
| clarkb | ya I think it does too, but your lower constraints should be publised for each python version iirc | 21:47 |
| fungi | yeah, some of the py36 and py37 jobs got pinned to ubuntu-bionic already, but possible some were missed | 21:47 |
| clarkb | whoch means there should be a working list for 3.6 and 3.8 | 21:47 |
| rosmaita | clarkb: you mean in the cinder l-c we should be distinguishing 3.6 and 3.8? | 21:48 |
| clarkb | rosmaita: yes mit probably already does | 21:48 |
| smcginnis | clarkb: Two sets of lower-constraints? | 21:49 |
| *** Goneri has quit IRC | 21:49 | |
| clarkb | no, one set with python version specific constraints | 21:49 |
| smcginnis | I suppose we could denote ;python_version=3.6 | 21:49 |
| smcginnis | Yeah | 21:49 |
| clarkb | we've already been doing that fir a while | 21:49 |
| smcginnis | That makes sense. | 21:49 |
| rosmaita | well, i guess i could take our current l-c file and append ;python_version=3.6 to each line | 21:50 |
| smcginnis | We can run py38 on bionic, right? We could test things that way. | 21:50 |
| clarkb | I think so? you shouldn't need to though, you can make a variant of the constraints job that runs on focal | 21:51 |
| clarkb | is the job timeout representative of the issue? | 21:51 |
| rosmaita | no | 21:52 |
| clarkb | is there a link to the issue then? | 21:52 |
| rosmaita | https://review.opendev.org/#/c/743080/ | 21:53 |
| rosmaita | there's a comment about what we tend to see when the job doesn't time out | 21:53 |
| clarkb | can you link to the job? I started there and it took me to a timeout job | 21:53 |
| rosmaita | looking | 21:53 |
| rosmaita | https://c8a82e6058c8d023a45b-f54a194f01fabbfbf1677f1f837bd08c.ssl.cf2.rackcdn.com/743080/9/check/openstack-tox-lower-constraints/b080b14/testr_results.html | 21:54 |
| clarkb | thanks | 21:54 |
| rosmaita | when it doesn't time out, we see these | 21:54 |
| rosmaita | but not the same jobs each time | 21:55 |
| rosmaita | i mean tests | 21:55 |
| smcginnis | And weird behavior from stestr that it prints out Captured stdout and Captured stderr for every individual test. | 21:56 |
| smcginnis | Anyway, several repos have this issue, so I think keeping this job on bionic for now is the best bet. | 21:56 |
| smcginnis | Then we can look at adding version flags to the lower-constraints. | 21:56 |
| smcginnis | Traditionally, we didn't have that in those constraints files. But we've done that for upper-constraints, so no reason we can't. | 21:57 |
| smcginnis | And then it's even more betterer for telling downstream what our expected ranges are. | 21:57 |
| fungi | the other thing which didn't really get tackled as part of this transition is that the normal python3.6 jobs should move to centos-8, because ubuntu-bionic is not an official tested runtime platform for victoria | 21:58 |
| fungi | python3.6 was kept as a tested python runtime for victoria because it's what's in centos-8, so continuing to test it on ubuntu-bionic makes very little sense | 21:59 |
| rosmaita | we could try smcginnis's patch specifying centos-8 nodes and see what happens | 22:00 |
| smcginnis | fungi: Is the node name centos-8? | 22:00 |
| fungi | yes | 22:00 |
| fungi | i honestly have no idea if anybody's even tested doing that | 22:00 |
| rosmaita | it will be interesting to see what happens | 22:00 |
| openstackgerrit | Sean McGinnis proposed openstack/openstack-zuul-jobs master: Switch py36 jobs to centos-8 https://review.opendev.org/751112 | 22:01 |
| fungi | because for ussuri we had python3.6 jobs since they were the default python3 on ubuntu-bionic | 22:01 |
| smcginnis | Pretty sure it fill fail a lot of places. I've seen a lot of bindep.txt files that don't specify rpms. | 22:01 |
| fungi | however it gets tricky, because stable/ussuri should almost certainly continue to test py36 on ubuntu-bionic, but we haven't previously made distro-specific python unit test jobs (at least not in any widespread manner) | 22:02 |
| sorrison | anyone around to help me with a zuul v3 config question? I have a job that inherits from devstack-tempest but I don't want it to run the multinode-bridge ansible role defined in the multinode job that is inherited from. Can I disable it somehow instead of reinventing the whole hierarchy | 22:02 |
| clarkb | looking at the subunit file it almost looks like its the test harness that may be failing? | 22:02 |
| clarkb | smcginnis: rosmaita basically oyu get a test failure that ran in 0 seconds and with no additional details | 22:02 |
| openstackgerrit | Sean McGinnis proposed openstack/openstack-zuul-jobs master: Switch py36 jobs to centos-8 https://review.opendev.org/751112 | 22:03 |
| clarkb | smcginnis: rosmaita: it might be worth running those tests without stestr and see if there is anything suspicous about the output | 22:03 |
| clarkb | sorrison: that role acts on particular ansible groups, I think if you drop those groups it will noop | 22:03 |
| clarkb | sorrison: that is how it noops on the single node jobs iirc | 22:04 |
| fungi | which is likely easier than having to fork an ancestral job and descent from that | 22:04 |
| *** rlandy has quit IRC | 22:04 | |
| clarkb | smcginnis: rosmaita: also maybe check if it is always those jobs that have trouble | 22:04 |
| sorrison | clarkb: Ah ok, thanks, I'll look into that | 22:04 |
| rosmaita | clarkb: well, we know it's something about the l-c, because our py38 unit tests are basically the same job, and they run fine in focal | 22:05 |
| sorrison | clarkb: I'm a bit of a ansible newbie, do you happen to know of some example you can point me to? | 22:05 |
| sorrison | I know I need to drop the switch and peers group, just not exactly sure how | 22:06 |
| openstackgerrit | Sean McGinnis proposed openstack/openstack-zuul-jobs master: Switch py36 jobs to centos-8 https://review.opendev.org/751112 | 22:06 |
| clarkb | sorrison: https://opendev.org/openstack/devstack/src/branch/master/.zuul.yaml compare single node and 2 node nodesets in that file | 22:08 |
| clarkb | basically oyu remove the switch and peers group entries | 22:08 |
| clarkb | rosmaita: I might even juts try something as simple as bumping python-subunit and testtools in lower-requirements for python3.8 | 22:10 |
| sorrison | great, thanks, so I basically define my own nodegroup with the same name | 22:10 |
| clarkb | sorrison: you might need to use a different name and tell the job to use the different name | 22:10 |
| clarkb | sorrison: but yes make a new nodeset and use that | 22:11 |
| sorrison | ok cheers | 22:11 |
| clarkb | sorrison: out of curiousity why not set up the shared networking ? | 22:12 |
| sorrison | clarkb: this is for a networking-midonet job, the vxlan interface that is created conflicts with midonet vxlan for some reason I don't understand | 22:14 |
| sorrison | essentially doing a `sudo ip l del vxlan_sys_4789` inside the multinode job fixed it so I was just trying to do it in a nicer way | 22:15 |
| clarkb | overlapping with the vxlan ids maybe? | 22:17 |
| clarkb | we do some large offset to avoid what neutron uses by default which is to start low and increment from there iirc | 22:17 |
| rosmaita | clarkb: bumped those both up to what's in u-c, let's see what happens | 22:18 |
| sorrison | clarkb: yeah maybe, I'm no networking expert but from what it looks like midonet tries to control it all, it expects to have free reign I think | 22:19 |
| clarkb | the problem that bridge is trying to solve is that dependingon the cloud our test nodes may get a /32 network or no ipv6 at all etc | 22:20 |
| clarkb | what that bridge does is create an l2 network across the hosts that can be used to work around that if necessary | 22:21 |
| clarkb | or if you just need an l2 network | 22:21 |
| *** vishalmanchanda has quit IRC | 22:21 | |
| sorrison | yeah ok, don't think we need an of that for this case | 22:21 |
| sorrison | I have access to a couple nodes put on hold for me which I've been testing with | 22:22 |
| *** zzzeek has quit IRC | 22:26 | |
| *** zzzeek has joined #openstack-infra | 22:27 | |
| *** Lucas_Gray has quit IRC | 22:35 | |
| *** sreejithp has quit IRC | 22:38 | |
| *** tosky has quit IRC | 22:42 | |
| *** gmann_afk is now known as gmann | 22:49 | |
| rosmaita | clarkb: no luck with bumping testtools and subunit | 22:50 |
| clarkb | rosmaita: was it the same tests? and any more debugging info with newer tools? | 22:50 |
| rosmaita | no, different tests but in that same general area | 22:52 |
| rosmaita | fungi: how do you feel about https://review.opendev.org/#/c/751107/1 ? | 22:54 |
| *** rcernin has joined #openstack-infra | 22:58 | |
| *** rcernin has quit IRC | 22:59 | |
| *** rcernin has joined #openstack-infra | 22:59 | |
| openstackgerrit | Merged openstack/openstack-zuul-jobs master: Run openstack-tox-lower-constraints on bionic https://review.opendev.org/751107 | 23:04 |
| rosmaita | works for me! | 23:07 |
| *** smarcet has quit IRC | 23:07 | |
| gmann | clarkb: fungi rosmaita smcginnis i think we should run l-c on Focal. replied no review -https://review.opendev.org/#/c/751107/1 | 23:17 |
| clarkb | gmann: yes that is what I was saying, but I think for now not doing so isn't a major risk | 23:17 |
| clarkb | since focal is newer we can expect newer deps there | 23:18 |
| gmann | it can break many projects where Focal fixes are merged and if those constraint re not present in Bionic | 23:18 |
| gmann | it can make all ongoing and already merged work back to bionic then | 23:19 |
| clarkb | gmann: we need constraints for both versions of python | 23:20 |
| clarkb | we've done that with upper constraints for many relases now iirc | 23:20 |
| gmann | you mean two jobs then | 23:20 |
| clarkb | not sure about lower | 23:20 |
| clarkb | maybe, I'm not sure if we tested uppet constraints with all of the python versions | 23:20 |
| clarkb | but we list them all | 23:20 |
| gmann | but default should be what distro we have right | 23:21 |
| gmann | and we can add another job if we want to add 3.6 constraints checks also | 23:21 |
| clarkb | yes, I think once the issue is sorted out switching back to 3.8 is sensible | 23:21 |
| clarkb | if we're deciding thats the primary platform (iirc 3.6 is still supported) | 23:22 |
| gmann | you mean cinder issue? | 23:22 |
| clarkb | gmann: yes, but they said it affects other projects too | 23:22 |
| gmann | i do not think so. only cinder and keystone are failing which i am debugging currently. many other projects ~200 repos have been merged to fixes and were running on focal but now back to bionic | 23:23 |
| gmann | and can fail many of newer added version are not on Bionic | 23:23 |
| clarkb | gmann: I was going off what I was told "21:42:31 smcginnis | There are a bunch of other repos failing now because of running on focal too." | 23:24 |
| *** zzzeek has quit IRC | 23:24 | |
| gmann | humm. as per my status in last email i listed the repos failing or under fix. and those are very few and more than 200 repo were fixed and working fine | 23:25 |
| gmann | heat has been fixed in morning too | 23:25 |
| openstackgerrit | Ghanshyam Mann proposed openstack/openstack-zuul-jobs master: Revert "Run openstack-tox-lower-constraints on bionic" https://review.opendev.org/751116 | 23:25 |
| gmann | clarkb: ^^ i am reverting it to avoid breaking the more things than solving/. | 23:26 |
| gmann | and let's fix cinder and keystone and other if there is any | 23:26 |
| clarkb | smcginnis: ^ any clarification on what the other repos are? | 23:27 |
| clarkb | gmann: do we know why l-c is failing? I looked at it briefly and it appears that subunit isn't recording anything | 23:27 |
| gmann | this is cinder and keystone in-progress fixes https://review.opendev.org/#/c/743080/ https://review.opendev.org/#/c/743117/11 | 23:27 |
| gmann | clarkb: keystone failing on some sql things and cinder facing some timeout in tests which is caused by one of the package seems | 23:28 |
| fungi | you've basically got a revolt on your hands because projects are dealing with a flood of changes for feature freeze, and to them getting features finished for what they've promised will be included in the release is more important than making sure their jobs test what we've said we'll be releasing platform support for | 23:28 |
| *** zzzeek has joined #openstack-infra | 23:29 | |
| clarkb | fwiw I do think there is something to the argument that lower constraints should be tested with the oldest python | 23:29 |
| fungi | so folks need to decide whether they want to release software which will only work on outdated platforms or software which has to punt features to a future cycle | 23:29 |
| clarkb | so in this particular case I don't think we're violating the ensur jobs test what we've said we'll release for | 23:29 |
| clarkb | and the correct way to address conflicts between lower values for python3.8 and 3.6 is to use env markers which upper constraints do | 23:30 |
| fungi | fair, the question i suppose is whether lower-constraints also indicates installation on an older system | 23:31 |
| gmann | we have fixed almost ~80% of the repo (till today) to make it work on Focal, not sure reverting it to Bionic is good or not breaking. | 23:31 |
| clarkb | gmann: right I think l-c is a bit of an exception is what I'm saying | 23:32 |
| clarkb | since its specifically there to test the oldest versions of thing we support | 23:32 |
| clarkb | for everything else I agree | 23:32 |
| gmann | oldest versions of thing on runtime distro version right? or any distro version ? | 23:33 |
| clarkb | gmann: the supported runtimes which are currently 3.6 and 3.8 | 23:33 |
| clarkb | previously it was just 3.6 so it was a bit simpler | 23:33 |
| gmann | i mean Distro version Ubuntu Focal or Bionic becasue 3.8 on bionic is different then 3.8 on Focal | 23:34 |
| fungi | well, "3.6" in ussuri meant ubuntu-bionic | 23:34 |
| clarkb | gmann: well in thiscase it would be centos8 3.6 and focal 3.8 as fungi pointed out | 23:34 |
| clarkb | for the current release | 23:34 |
| fungi | "3.6" in victoria supposedly means centos-8 | 23:34 |
| fungi | but jobs aren't currently set up to split py36 jobs for ussuri and victoria to different platforms | 23:35 |
| gmann | ok, we can discuss this how we should do but for Focal migration can we proceed with 3.8 and keep all fixed things and not reverting them ? | 23:35 |
| gmann | and we can make it clear how we should test l-c when two different python versions supported in testing runtime | 23:36 |
| gmann | but for now i thin we should not change what already done | 23:36 |
| clarkb | again in tis case I think this is a fix because smcginnis' argument that lower constraints should test the oldest python is a good one imo | 23:36 |
| clarkb | the job is there specifically to test that the old stuff still works | 23:37 |
| fungi | i'm willing to side with the qa team for now as the tc has theoretically delegated project-wide testing decisions to them, but ultimately concerns with their choice need to be taken up with the tc i guess | 23:37 |
| clarkb | thats fair | 23:37 |
| clarkb | gmann: can you communicate with them that this is just as important as theirfeatures ? | 23:38 |
| gmann | whom? project side | 23:38 |
| fungi | and if cinder wants a cinder-specific lower-constraints job which runs on bionic rather than using the standard one, then i'm for that too and they and qa can bring their respective concerns over each choice to the tc | 23:38 |
| clarkb | yes cinder and whoever else has broken l-c jobs | 23:38 |
| gmann | ok but for then we need to revert it and if we are not able to solve it on time then it is fair as fungi suggested to keep their job on bionic instead of moving back all | 23:40 |
| gmann | because now in-progress fixes started running on bionic and cannot be continued | 23:41 |
| clarkb | gmann: you can run the job on focal until it is ready | 23:41 |
| clarkb | (same thing the other way around as I mentioned in my commnt on your change) | 23:42 |
| gmann | let me add their l-c on bionic and then we revert this so that all other projects keep doing on focal and i can work on fixes in paralell | 23:42 |
| clarkb | it may require a bit of template expansion but is doable | 23:42 |
| fungi | and yes, when there's more time for rational debate i do think deciding what the expected target platform for a typical lower-constraints job should ideally be would make a good topic (forum cfp is open through monday!) | 23:48 |
| gmann | +1 | 23:50 |
| gmann | i think we should discuss runtime testing things itself so that we can decide things are doable also. | 23:51 |
| fungi | it could warrant clarifying in the pti, and also is worth talking about whether lower-constraints jobs are expected to be kept stable on stable branches or can be removed sooner in the interest of dropping ancient node types from our ci system | 23:51 |
| gmann | especially in current situation | 23:51 |
| fungi | gmann: as smcginnis indicated, 751116 could use at least a sentence of rationale for the revert added to the commit message. then i'll approve it | 23:55 |
| gmann | fungi: yeah adding bionic version of l-c on cinder and keystone side. I will update the commit msg | 23:56 |
| fungi | cool, thanks | 23:56 |
| *** tetsuro has joined #openstack-infra | 23:58 | |
| smcginnis | I do not think that should be reverted. This is an issue for a lot of folks, right when we should be locking things down and getting last minutes features landed. This was the absolute worst timing to switch the platform. The least we can do is still run lower-constraints on our lowest constraints. | 23:58 |
| *** zzzeek has quit IRC | 23:58 | |
| clarkb | smcginnis: supposedly its just keystone and cinder? | 23:59 |
| fungi | there are "a lot of folks" working on keystone and cinder | 23:59 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!