| *** hamalq has quit IRC | 00:56 | |
| *** sboyron has joined #opendev-meeting | 08:04 | |
| *** hashar has joined #opendev-meeting | 08:41 | |
| *** yoctozepto has quit IRC | 11:02 | |
| *** yoctozepto has joined #opendev-meeting | 11:03 | |
| *** hashar is now known as hasharLunch | 12:09 | |
| *** hasharLunch is now known as hashar | 13:17 | |
| *** frickler has quit IRC | 15:37 | |
| *** mordred has quit IRC | 15:37 | |
| *** irclogbot_2 has quit IRC | 15:37 | |
| *** frickler has joined #opendev-meeting | 15:37 | |
| *** irclogbot_3 has joined #opendev-meeting | 15:38 | |
| *** irclogbot_3 has quit IRC | 15:51 | |
| *** irclogbot_2 has joined #opendev-meeting | 15:52 | |
| *** hashar has quit IRC | 16:01 | |
| *** hashar has joined #opendev-meeting | 16:03 | |
| *** mordred has joined #opendev-meeting | 16:10 | |
| *** hashar is now known as hasharAway | 17:48 | |
| *** hamalq has joined #opendev-meeting | 18:18 | |
| ianw | hello, we'll have an infra meeting imminently | 18:59 |
|---|---|---|
| fungi | ohai | 19:00 |
| fungi | i hope the imminent meeting is eminent | 19:01 |
| ianw | #startmeeting infra | 19:01 |
| openstack | Meeting started Tue Mar 23 19:01:16 2021 UTC and is due to finish in 60 minutes. The chair is ianw. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:01 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:01 |
| *** openstack changes topic to " (Meeting topic: infra)" | 19:01 | |
| openstack | The meeting name has been set to 'infra' | 19:01 |
| ianw | #topic Annoucements | 19:01 |
| *** openstack changes topic to "Annoucements (Meeting topic: infra)" | 19:01 | |
| fungi | this just in: clark takes a week off | 19:01 |
| ianw | i also spelt that wrong | 19:02 |
| fungi | nothing wrong with an annoucement or two | 19:02 |
| ianw | anyway, yes, no other global announcements | 19:02 |
| ianw | #topic Actions from last meeting | 19:02 |
| *** openstack changes topic to "Actions from last meeting (Meeting topic: infra)" | 19:02 | |
| ianw | #link http://eavesdrop.openstack.org/meetings/infra/2021/infra.2021-03-16-19.01.html minutes from last meeting | 19:03 |
| ianw | we didn't seem to have any particular action items | 19:03 |
| ianw | #topic Specs approval | 19:03 |
| *** openstack changes topic to "Specs approval (Meeting topic: infra)" | 19:03 | |
| fungi | is the gerrit server replacement spec ready for consideration? | 19:04 |
| ianw | not quite, i was going to start up a new server and then fill in some things from info from that | 19:04 |
| fungi | cool. i'm good with what's there so far anyway | 19:05 |
| ianw | #topic Priority Efforts | 19:05 |
| *** openstack changes topic to "Priority Efforts (Meeting topic: infra)" | 19:05 | |
| ianw | #topic Update Config Management | 19:05 |
| *** openstack changes topic to "Update Config Management (Meeting topic: infra)" | 19:05 | |
| ianw | I think we will cover the active parts of this in other topics | 19:06 |
| ianw | #topic Opendev | 19:06 |
| *** openstack changes topic to "Opendev (Meeting topic: infra)" | 19:06 | |
| ianw | the main work here is the Gerrit account inconsistencies | 19:06 |
| ianw | this is really being driven by clarkb, but maybe fungi you have a update? | 19:07 |
| fungi | nothing new this week, no | 19:07 |
| fungi | in some belated afs news, the debian 10.9 stable point release will include the awaited openafs fix | 19:07 |
| fungi | so we should be able to simplify our buster image builds next week | 19:07 |
| fungi | release is scheduled for saturday | 19:08 |
| ianw | cool. executors rely on AFS from outside the container though, right? | 19:08 |
| fungi | yes, in our case i believe so | 19:09 |
| fungi | but this was for testing it | 19:09 |
| fungi | where we added the temporary workaround | 19:09 |
| fungi | or at least that's the only lingering workaround i remember | 19:10 |
| ianw | ++ | 19:10 |
| fungi | also i've just about hammered out getting zuul-jobs working with our gentoo images again, thanks to prometheanfire's help | 19:10 |
| ianw | yeah i saw something fly by -- i feel like gentoo images are currently not building | 19:12 |
| fungi | oh, again? i'll check that too | 19:12 |
| ianw | that was something to do with iscsi and newer gcc | 19:12 |
| fungi | they were working a few days ago | 19:12 |
| fungi | ahh, right, that. i think he had a fix happening upstream there | 19:12 |
| fungi | and for simplifying our gerrit all-projects acl, i looked into repurposing the openstack/openstack acl to contain the openstack release management bits, but ultimately determined that was a non-starter due to an exclusive setting in one section. so i've tentatively settled on making "openstack/meta-config" as the empty project for other openstack projects to inherit, but am not thrilled with the name | 19:13 |
| fungi | (especially considering we may want to recommend this model to other namespaces) | 19:13 |
| ianw | ok, i've noticed failures on some glean changes i've pushed, will have to look closer | 19:13 |
| fungi | i'll push the change up for openstack/meta-config later today, folks can follow up there if they have good name suggestions | 19:14 |
| ianw | ok, this is for release managers to remove old branches? | 19:14 |
| fungi | well, more generally, to get openstack release manager permissions out of our global config and into an openstack-only acl | 19:15 |
| fungi | so that, e.g., openstack release managers can't accidentally push tags for airship | 19:16 |
| ianw | ahh, right, got it | 19:16 |
| fungi | but yes also so that they can't accidentally delete another project's branches | 19:17 |
| ianw | there's also a note in the agenda about configuration tuning | 19:18 |
| fungi | ahh, yup | 19:18 |
| ianw | i'm not sure we've discussed that previously | 19:18 |
| fungi | oh, also on the gerrit theme, i've pushed up some changes to partially restore launchpad in-progress integration | 19:19 |
| fungi | as a stopgap until someone has time to write the replacement | 19:19 |
| fungi | #link https://review.opendev.org/782538 Stop trying to assign Launchpad bugs | 19:19 |
| fungi | #link https://review.opendev.org/782540 Run update-bug on patchset-created again | 19:20 |
| ianw | ++ that seems like a good compromise | 19:20 |
| fungi | the first one seems to have a job failure, likely bitrot for jeepyb | 19:20 |
| fungi | i'll look at it shortly | 19:20 |
| fungi | oh, and we're on a new version of zuul (4.1.0) but had to roll back off master temporarily | 19:21 |
| fungi | corvus has fixed the bug we rolled back for, and we'll be restarting again on latest master shortly after this meeting | 19:21 |
| corvus | and swest fixed the next bug we would have seen which avass found :) | 19:22 |
| *** sboyron has quit IRC | 19:22 | |
| corvus | (2nd bug only affected github; we probably would have seen it eventually) | 19:22 |
| fungi | i also revisited the gerrit upgrade fallout pad and tried to catch it up to current reality | 19:23 |
| ianw | ok, will watch out for all that and any new behaviour | 19:23 |
| fungi | #link https://etherpad.opendev.org/p/gerrit-3.2-post-upgrade-notes | 19:23 |
| corvus | however, i think we had a "pretty good" run on 4.1.0 in that the openstack tenant was fully running with the event queues in zk, with, afaict, no appreciable change in performance or load on zk. so i'm not too worried about the switch back. | 19:23 |
| fungi | if there's anything still in there which we've fixed or can stop worrying about, please mark it off the list | 19:24 |
| ianw | #link https://grafana.opendev.org/d/5Imot6EMk/zuul-status?orgId=1 | 19:25 |
| ianw | for anyone who hasn't seen recent updates to add zookeeper stats in there | 19:25 |
| fungi | oh, also there's a push to get debian-bullseye images added, starting with package mirroring. i think we'll need to evaluate quota usage on that volume as well as afs01.dfw overall | 19:26 |
| ianw | i think i may still owe some cleanups on fedora | 19:26 |
| fungi | checking out the volume utilization on our afs stats grafana dashboard, quite a few volumes are almost full, yeah | 19:27 |
| ianw | #link https://grafana.opendev.org/d/T5zTt6PGk/afs?orgId=1 | 19:27 |
| fungi | i suggested seeing if we can drop debian-stretch mirroring, but a number of openstack projects are still etsting with it on older stable branches | 19:27 |
| ianw | the wheel release stats there are depressing. i'll have to look at that | 19:27 |
| fungi | though related, we still have a node label named "debian-stable" aliasing stretch, when buster is the current stable as of a couple of years ago | 19:28 |
| fungi | we should probably encourage people to reevaluate their use of that, and either update or remove it | 19:29 |
| ianw | we do have plenty of disk quota in rax dfw so adding a drive to vicepa might be the simplest thing | 19:30 |
| fungi | yeah, though the more cinder volumes we attach the more precarious it becomes, as we saw with the old static.o.o | 19:31 |
| fungi | we're basically multiplying the odds of catastrophic failure by the number of volumes | 19:31 |
| ianw | or even afs01.dfw, when i rebooted it recently :) | 19:31 |
| ianw | one thing i've been meaning to look at too, after that OVH region burnt down, is the redundancy of tarballs in particular | 19:32 |
| fungi | in theory we replicate that, and can turn a read-only replica into the new read-write replica | 19:32 |
| ianw | it's sort of related to the failure mode; when we have vos release failures and require full releases, we get tied up in days and days of copying | 19:33 |
| fungi | yup | 19:33 |
| fungi | as for the recent afs01.dfw boot failure, i'm almost certain it's because we created the pv on the raw vilume block device and not a partition | 19:34 |
| fungi | i have a feeling we could reproduce that if we wanted | 19:34 |
| ianw | still, since we moved to running releases via ssh i think things have generally been more reliable | 19:34 |
| fungi | yes, that's helped immensely | 19:34 |
| ianw | we also spent quite a long time diagnosing and tuning rsync to stop touching every file for some updates too, which helped | 19:35 |
| ianw | alright, i think let's move on | 19:36 |
| ianw | #topic General Topics | 19:36 |
| *** openstack changes topic to "General Topics (Meeting topic: infra)" | 19:36 | |
| ianw | #topic Puppet/Ansible rewrites | 19:36 |
| *** openstack changes topic to "Puppet/Ansible rewrites (Meeting topic: infra)" | 19:36 | |
| ianw | i think the news of the week here was the launchers all switched over to fresh opendev.org versions | 19:36 |
| ianw | i think that leaves zuul scheduler host as the only Xenial system in that ecosystem? | 19:36 |
| ianw | executors, mergers, builders and launchers are all done now | 19:37 |
| fungi | zk servers? | 19:38 |
| ianw | i'm guessing with the pace of zuul development at the moment, we're better waiting a little to tackle that host | 19:38 |
| fungi | yeah, just double-checked, our zk servers are also still xenial | 19:38 |
| fungi | we should be able to rolling-replace those live | 19:38 |
| fungi | though as corvus observed, doing that will end in zuul only connecting to two out of the three until the next time the zuul services are restarted | 19:39 |
| fungi | because it won't automatically redistribute connections, only reconnect as needed | 19:40 |
| ianw | i'm willing to help out on that, a good way to become more familiar with zk operations | 19:40 |
| ianw | corvus: ^ maybe reach out when it's a better time to consider this, i.e. not pending bug restarts for bug fix updates :) | 19:41 |
| ianw | #link https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades | 19:41 |
| ianw | i had a quick pass through that | 19:41 |
| fungi | i'm tempted to snapshot the wiki server and try an in-place ubuntu upgrade for now, as repugnant as that idea may be | 19:42 |
| fungi | part of why the wiki server isn't listed there is that it's not running xenial. still on trusty :/ | 19:43 |
| corvus | i think we can upgrade zk any time; it's containerized, so we should already be running a recent release of the software; hopefully an os upgrade won't have too big of an impact | 19:43 |
| corvus | ianw: i think if you want to go ahead and stage the patches to do that, it's probably okay to do so more or less any time | 19:44 |
| ianw | corvus: ok, i'll take a look and see what i come up with | 19:44 |
| ianw | i feel like clarkb might have already written a change to switch to focal in testing at least | 19:45 |
| fungi | that does sound familiar | 19:45 |
| ianw | one from that list was the asterisk server; i feel like retirement is probably the best idea there | 19:45 |
| ianw | do we want a spec, or an announcement, or just changes we can vote on? | 19:46 |
| fungi | announcement is probably in order, just in case anyone was using it | 19:47 |
| fungi | ideally we'd work out how to move the current dial-in trunk's sip config to meetpad, but that's not absolutely necessary | 19:47 |
| ianw | openstack-discuss or just the service list? | 19:47 |
| fungi | i'd say service-announce | 19:48 |
| ianw | ok, i'll give myself an action item to get that going | 19:48 |
| fungi | thanks! | 19:48 |
| ianw | #action ianw start retirement for asterisk | 19:49 |
| ianw | there's nothing else on that list that is a surprise ... just a bunch of things we know we need to do :) but it is getting smaller | 19:49 |
| ianw | #topic Refstack | 19:50 |
| *** openstack changes topic to "Refstack (Meeting topic: infra)" | 19:50 | |
| ianw | speaking of, i think this is almost ready to be dropped as a topic | 19:50 |
| ianw | i have one outstanding bugfix review | 19:50 |
| fungi | excellent | 19:50 |
| ianw | #link https://review.opendev.org/c/opendev/system-config/+/781593 | 19:50 |
| ianw | i will put in a todo to clean up the old server in a few months just to be super safe | 19:51 |
| ianw | otherwise, i'd say this one is done | 19:51 |
| fungi | yay! | 19:52 |
| ianw | #topic PTG planning | 19:52 |
| *** openstack changes topic to "PTG planning (Meeting topic: infra)" | 19:52 | |
| ianw | last week clarkb put out a call for suggestions on this, did we want dedicated times to talk, or a hackathon, etc | 19:52 |
| ianw | tbh i feel like pretty much every day is a hackathon :) | 19:52 |
| fungi | yeah, it's more like do we want a hackathon where we're all awake at the same relatively inconvenient time ;) | 19:53 |
| ianw | i feel like the requests for times deadline was this thursday? | 19:54 |
| fungi | anyway, i gave my loose suggestions last week, don't really have any new ideas personally | 19:54 |
| fungi | yeah, maybe i'll double-check the ethercalc and see if he reserved anything | 19:55 |
| fungi | amusing side-note, the ptg organizers forgot we run an ethercalc instance and created a spreadsheet on the ethercalc.org site instead, which has been going up and down and returning random errors to people | 19:56 |
| ianw | ok, maybe i'll send a mail too. just in case anyone who doesn't hang out in meetings has an interest | 19:56 |
| fungi | thanks | 19:56 |
| ianw | it would certainly be worth it if we have a dedicated time to help onbaord people who are interested, etc. | 19:57 |
| ianw | #topic Open Discussion | 19:57 |
| *** openstack changes topic to "Open Discussion (Meeting topic: infra)" | 19:57 | |
| fungi | yes, especially new config reviewers | 19:57 |
| fungi | but anybody really | 19:57 |
| ianw | this is true, it is probably worth reserving a time dedicated for that, see who turns up | 19:58 |
| ianw | there's been a bit of work on glean lately if anyone wants to look | 19:58 |
| fungi | #link https://ethercalc.net/oz7q0gds9zfi PTG schedule spreadsheet | 19:58 |
| fungi | i don't see opendev reserving any slots in there yet | 19:58 |
| ianw | basically all open changes. ironic have some requirements there | 19:59 |
| fungi | oh, following up on the gentoo image builds, prometheanfire has a dib change proposed to solve it | 19:59 |
| fungi | see #opendev for details | 19:59 |
| ianw | ok will look | 20:00 |
| fungi | thanks for chairing, ianw! | 20:00 |
| *** hashar_ has joined #opendev-meeting | 20:00 | |
| ianw | that's about time, see you all next time! | 20:00 |
| ianw | #endmeeting | 20:00 |
| *** openstack changes topic to "Incident management and meetings for the OpenDev sysadmins; normal discussions are in #opendev" | 20:00 | |
| openstack | Meeting ended Tue Mar 23 20:00:27 2021 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 20:00 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/infra/2021/infra.2021-03-23-19.01.html | 20:00 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/infra/2021/infra.2021-03-23-19.01.txt | 20:00 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/infra/2021/infra.2021-03-23-19.01.log.html | 20:00 |
| *** hasharAway has quit IRC | 20:01 | |
| *** hashar__ has joined #opendev-meeting | 20:03 | |
| *** hashar_ has quit IRC | 20:06 | |
| *** hashar__ has quit IRC | 20:35 | |
| *** hamalq has quit IRC | 23:55 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!