| *** openstack has joined #opendev-meeting | 07:30 | |
| *** ChanServ sets mode: +o openstack | 07:30 | |
| *** hamalq has joined #opendev-meeting | 16:27 | |
| *** hamalq has quit IRC | 16:28 | |
| *** hamalq has joined #opendev-meeting | 16:29 | |
| *** hamalq has quit IRC | 17:16 | |
| *** hamalq has joined #opendev-meeting | 17:16 | |
| clarkb | meeting time now | 19:00 |
|---|---|---|
| clarkb | we'll get started in a minute or two | 19:00 |
| ianw | o/ | 19:01 |
| clarkb | #startmeeting infra | 19:01 |
| openstack | Meeting started Tue Jul 21 19:01:33 2020 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:01 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:01 |
| *** openstack changes topic to " (Meeting topic: infra)" | 19:01 | |
| *** zbr|rover has joined #opendev-meeting | 19:01 | |
| openstack | The meeting name has been set to 'infra' | 19:01 |
| zbr|rover | o/ | 19:01 |
| corvus | o/ | 19:02 |
| clarkb | #link http://lists.opendev.org/pipermail/service-discuss/2020-July/000058.html Our Agenda | 19:02 |
| clarkb | #topic Announcements | 19:02 |
| *** openstack changes topic to "Announcements (Meeting topic: infra)" | 19:02 | |
| clarkb | OpenDev virtual event #2 happening July 20-22 | 19:03 |
| clarkb | This event is happening. The upgraded version of etherpad seems to be holding up just fine | 19:04 |
| clarkb | #topic Actions from last meeting | 19:04 |
| *** openstack changes topic to "Actions from last meeting (Meeting topic: infra)" | 19:04 | |
| clarkb | #link http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-07-14-19.01.txt minutes from last meeting | 19:04 |
| clarkb | No actions recorded | 19:05 |
| clarkb | #topic Specs approval | 19:05 |
| *** openstack changes topic to "Specs approval (Meeting topic: infra)" | 19:05 | |
| clarkb | #link https://review.opendev.org/#/c/731838/ Authentication broker service | 19:05 |
| clarkb | again not quite ready for approval, but definitely available for feedback | 19:05 |
| clarkb | fungi: ^ anything else to add on this? You saw my feedback? | 19:06 |
| fungi | yep, thanks for reviewing that, will incorporate into the next revision | 19:09 |
| clarkb | #topic Priority Efforts | 19:10 |
| *** openstack changes topic to "Priority Efforts (Meeting topic: infra)" | 19:10 | |
| clarkb | #topic Update Config Management | 19:10 |
| *** openstack changes topic to "Update Config Management (Meeting topic: infra)" | 19:10 | |
| clarkb | Zuul and Nodepool are running on containers now with the exception of nb03 | 19:11 |
| clarkb | nb03 runs on an arm64 host so needs an arm64 image | 19:12 |
| clarkb | I've collected a number of arm64 container image build fixes for nodepool in this change https://review.opendev.org/#/c/741942/ | 19:12 |
| clarkb | latest problem is that cryptography did a release between the time we had working wheel cache setup on cryptography 2.9.2 and vhd-util ppa being fixed so now there is an uncached cryptography 3.0.0 and we try to build that on buildx and timeout | 19:13 |
| corvus | ping me if you need me here; i'm working on zuul emergency restart | 19:13 |
| clarkb | ianw: ^ we expect that mirror to update periodically right? any idea how soon we should have a new wheel for cryptography? | 19:13 |
| clarkb | but I think ocne that happens we can land that change then we will have an image and can start looking at updating nb03 | 19:14 |
| ianw | yes, wheels *should* update daily, however, i had a look and there were network issues the other day | 19:15 |
| ianw | kevinz fiddled something with routers in the cloud, and i haven't looked since. so i'll get back to it and see -- but the jobs couldn't get a reliable ipv4 connection and so afs was just not happy | 19:16 |
| clarkb | got it, in any case that seems to be the current issue and so getting the wheel cache updated would be helpful | 19:16 |
| clarkb | Related to services running in containers we've just discovered that docker-compose up -d doesn't pull new images if it already has images but they are out of date | 19:17 |
| clarkb | something to be aware of and maybe something we'll address in our playbooks | 19:17 |
| ianw | one thing is that the release job only releases if *all* the wheel builds pass ... i'll look at that. it might have just been done like that before we could do something like a semaphore | 19:17 |
| clarkb | ianw: if we're able to safely copy out successful builds doing partial updates seems fine? | 19:17 |
| clarkb | I mean if we don't have new A or B package we'll try to build both in the downstream jobs. If we have only A or B, we'll try to build the other and that is still a net win time wise | 19:18 |
| clarkb | if we want different versions that needs to be controlled by requirements or constraints | 19:18 |
| ianw | yeah, there's no comments that suggest it was done for particular consistency reasons | 19:18 |
| clarkb | http://lists.openstack.org/pipermail/openstack-discuss/2020-July/016022.html is also related to updating config management | 19:20 |
| clarkb | I'ev sent a response and approved on of the changes. But more input on the general goal there would be good. The mailing list would probably be best for that so that Javier sees it too though | 19:20 |
| clarkb | #topic OpenDev | 19:21 |
| *** openstack changes topic to "OpenDev (Meeting topic: infra)" | 19:21 | |
| clarkb | #link http://lists.opendev.org/pipermail/service-discuss/2020-July/000057.html Advisory Board welcomed. | 19:21 |
| clarkb | I think that mostly makes the advisory board and its membership official | 19:22 |
| clarkb | I've encouraged our volunteers to get involved and we'll see where it takes us | 19:22 |
| fungi | ianw: possibly related to network issues, but sslcheck is also saying for the past couple of days that the mirror cert there is is expiring unexpectedly soon | 19:22 |
| ianw | fungi: ok, will look | 19:22 |
| clarkb | #link http://lists.opendev.org/pipermail/service-announce/2020-July/000007.html Gerrit /p/ mirror deprecation. | 19:22 |
| clarkb | This is another email I wrote. With plan to get ahead of gerrit repurposing /p/ for project dashboards instead of git mirrors | 19:23 |
| clarkb | this simplifies the number of locations we need to replicate and makes branch management simpler | 19:23 |
| clarkb | and since it will happen when we update gerrit anyway we may as well do it now and get those other wins :) | 19:23 |
| clarkb | I'll propose a change to make those urls 403 when I get a momement, but I have about a week and a half | 19:24 |
| clarkb | and I'll go ping the cinder channel too as they seem to be the bulk of the users of that old path | 19:25 |
| clarkb | Anything else OpenDev related to bring up? | 19:25 |
| clarkb | #topic General Topics | 19:26 |
| *** openstack changes topic to "General Topics (Meeting topic: infra)" | 19:26 | |
| fungi | the openstack-infra ml is now closed | 19:26 |
| * fungi wasn't quick enough | 19:26 | |
| clarkb | fungi: oh ya, thank you for doing that | 19:26 |
| fungi | np | 19:26 |
| clarkb | #topic Bup and Borg Backups | 19:26 |
| *** openstack changes topic to "Bup and Borg Backups (Meeting topic: infra)" | 19:26 | |
| clarkb | ianw: for Bup I was curious if you had success doing a restore from the hosts that had their local indexes cleaned up. | 19:27 |
| clarkb | and for Borg I think it would probably be good to have a short intro to your thoughts/plan around it | 19:27 |
| ianw | ahhh, yes, i didn't get to that restore. i will loop back on that | 19:28 |
| ianw | mostly because i started looking at a new backup server, and it seems, as we've discussed before, bup is effectively dead | 19:28 |
| ianw | there's no python3 support, and it doesn't seem to be coming unfortunately, and it's dropped from focal | 19:29 |
| ianw | so i had a serious look at rdiff-backup and borg; both are very close to bup in terms of the hosts backing themsevles up to a central location and deduping on the server side | 19:29 |
| ianw | borg seemed to be more active, had clarkb's approval, and had a lot of other features | 19:30 |
| fungi | the clarkb seal of approval | 19:30 |
| ianw | #link https://review.opendev.org/741366 | 19:30 |
| clarkb | (ya I use borg locally, I've not had to use it in an emergency yet but seems to have a number of good features like append only support, encryption at rest if we want it, fuse mounts for restoration) | 19:30 |
| ianw | so that is the roles to do borg backups | 19:30 |
| ianw | i'd suggest, if we're happy to move on with it, we bring up a borg backup server and switch in a ... lesser value host (maybe not start with review i mean) ... and start from there to get a bit of experience | 19:31 |
| clarkb | I like that idea | 19:32 |
| fungi | seems we're backing up review-dev | 19:32 |
| clarkb | also we potentially get the ability to prune append only backups? | 19:32 |
| fungi | or we were anyway | 19:32 |
| clarkb | not sure how effective that will be but that could be a great feature | 19:32 |
| ianw | append-only is implemented in above -- it does seem that we could run jobs on the backup server to then prune repositories | 19:33 |
| ianw | so clients can't delete their history (restricted ssh command), but we can on the server side | 19:33 |
| clarkb | and in reviewing the change this is implemented completely to the side of bup so we could switch a host over with little pain I expect? | 19:35 |
| clarkb | basically add it to borg backups, let it backup with borg, check things, then disable bup ? | 19:35 |
| ianw | yes, i deliberately kept them totally separate, with separate group names etc. so they can run completely in parallel | 19:35 |
| fungi | thanks, wise plan | 19:36 |
| ianw | that's more or less it; if we like it, i'm happy to move on with starting the server and initial backup runs (as mentioend eview-dev seems a good place to start) | 19:37 |
| corvus | ++ | 19:38 |
| clarkb | I think I'm ok with it, but corvus would be a good source of feedback too having built the bup stuff | 19:39 |
| clarkb | I'll review the latest ps looks like some of my feedback was addressed | 19:39 |
| clarkb | #topic Project Renames | 19:39 |
| *** openstack changes topic to "Project Renames (Meeting topic: infra)" | 19:39 | |
| clarkb | Last week we pencilled in Friday for project renames. I think I'm going to be in a reasonable spot to do that still. fungi are you still able to help? | 19:40 |
| fungi | i am indeed | 19:40 |
| fungi | still wide open for friday | 19:40 |
| clarkb | great, should we say ~15:00 UTC on friday for that? I've been having ealry days with opendev event happening so keeping to that schedule seems fine | 19:40 |
| clarkb | and thursday ish we can curate an etherpad and the chagnes and make sure we are ready | 19:41 |
| fungi | i'm good with 1500z | 19:41 |
| clarkb | great. I can sent an email about that to service-announce today after lunch | 19:42 |
| clarkb | #topic Server Upgrades | 19:43 |
| *** openstack changes topic to "Server Upgrades (Meeting topic: infra)" | 19:43 | |
| clarkb | fungi: anything new on wiki side of things? I don't expect so but didn't want to leave it out if there is | 19:43 |
| fungi | zilch | 19:44 |
| clarkb | #topic Open Discussion | 19:44 |
| *** openstack changes topic to "Open Discussion (Meeting topic: infra)" | 19:44 | |
| clarkb | That was it for our scheduled agenda. Anything else to bring up? | 19:44 |
| clarkb | I guess I should mention that https://review.opendev.org/#/c/741277/ is the next thing in my series of branch management changes. Land that should be safe and it won't be used until we tag the library. https://review.opendev.org/741279 tests it | 19:45 |
| clarkb | Sounds like that may be it? Thank you everyone. I know it was a somewhat distracted meeting. See you next week. | 19:49 |
| clarkb | #endmeeting | 19:49 |
| *** openstack changes topic to "Incident management and meetings for the OpenDev sysadmins; normal discussions are in #opendev" | 19:49 | |
| openstack | Meeting ended Tue Jul 21 19:49:24 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 19:49 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-07-21-19.01.html | 19:49 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-07-21-19.01.txt | 19:49 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-07-21-19.01.log.html | 19:49 |
| fungi | thanks clarkb! | 19:49 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!