| *** pleimer has joined #opendev-meeting | 13:14 | |
| *** pleimer_ has joined #opendev-meeting | 13:53 | |
| *** pleimer has quit IRC | 13:59 | |
| clarkb | Anyone else here for the meeting? | 19:00 |
|---|---|---|
| clarkb | we'll get started shortly | 19:00 |
| ianw | o/ | 19:00 |
| clarkb | #startmeeting infra | 19:01 |
| opendevmeet | Meeting started Tue Jun 8 19:01:16 2021 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:01 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:01 |
| opendevmeet | The meeting name has been set to 'infra' | 19:01 |
| clarkb | #link http://lists.opendev.org/pipermail/service-discuss/2021-June/000253.html Our Agenda | 19:01 |
| clarkb | The agenda isn't very big today. Hopefully I'll be able to give you some time back | 19:01 |
| clarkb | #topic Announcements | 19:01 |
| clarkb | This didn't make the agenda, but it occurred to me wheil prepping for this meeting. I've got family in town starting tonight and I may not be able to run the next two meetings. I think I'll be around next week but definitely not the week after | 19:02 |
| clarkb | I'll try to figure that out more accurately once i can discuss their plans tonight/tomorrow | 19:02 |
| clarkb | #topic Actions from last meeting | 19:03 |
| clarkb | #link http://eavesdrop.openstack.org/meetings/infra/2021/infra.2021-06-01-19.01.txt minutes from last meeting | 19:03 |
| clarkb | There were no recorded actions | 19:03 |
| clarkb | #topic Specs | 19:04 |
| clarkb | #link https://review.opendev.org/c/zuul/zuul/+/793669 Zuul spec to use matrix instead of irc | 19:04 |
| clarkb | This isn't our spec but it is worth keeping an eye on and commenting on if you have input | 19:04 |
| fungi | i think corvus is hoping to make the decision on it later this week | 19:04 |
| fungi | so feedback sooner would be better | 19:05 |
| clarkb | ++ | 19:05 |
| clarkb | #topic Topics | 19:05 |
| clarkb | #topic Switch to OFTC | 19:05 |
| clarkb | ianw has been working to redo our irc bot deployment stuff. Including spinning up a new server and running bots out of containers there | 19:06 |
| fungi | and a new eavesdrop server | 19:06 |
| clarkb | meetbot and ptgbot are the bots left out in that process as they aren't currently containerized | 19:06 |
| fungi | er, you did say new server, yeah | 19:06 |
| clarkb | #link https://review.opendev.org/q/topic:%22limnoria%22+status:open Limnoria bot rewrite | 19:06 |
| clarkb | This topic is a set of changes that converts meetbot from supybot to limnoria and containerized that bot so ti can run on the new server too | 19:07 |
| fungi | statusbot, btw, is still running foreground in a screen session, hasn't crashed again :/ | 19:07 |
| clarkb | I think I've gotten through that topic for reviews but should double check | 19:07 |
| clarkb | #link https://review.opendev.org/c/openstack/ptgbot/+/793792 Update PTGbot for OFTC | 19:07 |
| clarkb | This change updates ptgbot to make it possible for it to connect to oftc, but we need to double check if it needs python2 or if it can run under python3. If it can run under python3 then a good followup would be containerizing it. If not then converting to python3 then containerizing it would be good | 19:07 |
| fungi | oh also, i think the openstack tc has decided friday of this week is when they'll change channel topics on freenode, not sure if anyone else is following suit or if we want to do something similar for our handful of opendev channels there | 19:08 |
| ianw | so the bits that want review now are the deployment | 19:08 |
| ianw | #link https://review.opendev.org/q/topic:%22meetbot-container%22+(status:open%20OR%20status:merged) | 19:08 |
| clarkb | ah ok I'll have to review those then | 19:09 |
| ianw | i am still working on statusbot in | 19:09 |
| ianw | #link https://review.opendev.org/c/opendev/system-config/+/795213 | 19:09 |
| clarkb | fungi: I'm happy to update our topic for #opendev and #opendev-meeting though I dropped off of freenode today. Neither channel had any real traffic since the switc hthough | 19:09 |
| ianw | that's the only bit i would consider hacky. i'm installing it in the limnoria container, even though it's not a plug-in, on the theory that it *should* be | 19:09 |
| fungi | yeah, i can do topic updates, i'm still joined there | 19:10 |
| ianw | but realistically, i doubt anyone is that interested in rewriting it; but if they did, that's where it should go | 19:10 |
| clarkb | ianw: you mean rather than using a separate statusbot container? | 19:10 |
| clarkb | ianw: is there a reason to combine them rather than do process per container? I think we have tried to stick to process per container as much as possible | 19:11 |
| clarkb | I'll try to dig into that when get to reviewing the changes | 19:13 |
| ianw | well, to avoid having to make another container and build process for it all, for something that we don't really care that much about | 19:13 |
| clarkb | I see | 19:13 |
| clarkb | Another thing I wanted to point out is that zuul is using a matrix element oftc connection url instead of oftc's webchat as easy connect mode. If we've got similar links that need updating I think that worked reasonable well | 19:14 |
| clarkb | might also suggest it to other users like openstack | 19:14 |
| clarkb | anything else on the subject of IRC? | 19:15 |
| ianw | i guess ptgbot i wanted to call out | 19:16 |
| ianw | i've -1'd | 19:16 |
| ianw | #link https://review.opendev.org/c/opendev/puppet-ptgbot/+/793794 | 19:17 |
| fungi | we have time to solve ptgbot, there's no looming ptg for months thankfully | 19:17 |
| ianw | explaining that the puppet no longer runs on eavesdrop01.openstack.org (because we're worried pip will explode on itself) | 19:17 |
| clarkb | ianw: oh I didn't realize there was a puppet change already too. I agree we should convert it over. Particularly since we have time for that bot | 19:17 |
| fungi | so if it's offline for a while to get containerified, it's not a huge deal | 19:17 |
| ianw | so the future for this, if there is any, is for someone to make a container and migrate it to ansible, and deploy on eavesdrop01 | 19:18 |
| clarkb | ++ | 19:18 |
| ianw | at one time, per the irc spec, the future was to rewrite it as a limnoria plugin | 19:18 |
| ianw | but i honestly don't think anyone is going to do that now | 19:18 |
| fungi | i have no idea how much of a forklift that would actually be, i expect the internal logic is mostly isolated and would probably not be that hard to translate to a limnoria plugin, but i can't say i've looked at it with that in mind | 19:19 |
| ianw | and i would not be surprised if it prompts a think about it's future anyway | 19:19 |
| clarkb | fungi: on the flip side I think installing it to a container and bind mounting the dir where it stores info is probably super trivial | 19:20 |
| fungi | yep | 19:20 |
| fungi | we already just pip install it anyway | 19:20 |
| corvus | o/ | 19:21 |
| clarkb | we can bring it up with diablo_rojo_phone and ttx to see what they think and take it from there | 19:21 |
| clarkb | I agree with ianw though taht we shouldn't try to keep the puppetry going given what hapepned with pip on the old server when we switched | 19:21 |
| ianw | i doubt containerising it would be too difficult. i'd just like someone else to own it :) | 19:22 |
| clarkb | ++ | 19:22 |
| clarkb | I suspect diablo_rojo_phone and/or ttx would be happy to help with that | 19:22 |
| clarkb | Lets move on | 19:22 |
| clarkb | #topic Gerrit Updates | 19:22 |
| ianw | i'm happy to help point people in the right direction to get it all going | 19:22 |
| clarkb | ianw: I can let them know. I can probably help too | 19:23 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/775961 MariaDB container for review02 accountPatchReviewDB | 19:23 |
| clarkb | ianw has been working on ^ as one of the last pieces before we can schedule a cut over to review02 | 19:23 |
| clarkb | I found one more thing reviewing it this morning that would be good to double check | 19:23 |
| ianw | ok i'll loop back. | 19:24 |
| clarkb | But otherwise I think it is just about there. It would be good if other infra-root can review it too since it has the potential to make review01 unhappy (though the change attempts to avoid changing anything on that) | 19:24 |
| clarkb | I also remembered that I had achange up to unfork gerrit | 19:24 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/791995 Stop adding special x/ path handling to gerrit | 19:24 |
| fungi | yeah, that's fixed and backported to the minor series we're on | 19:25 |
| clarkb | It passes testing, though we may want to double check we don't need to update plugin versions on the image build? | 19:25 |
| clarkb | Applying that will require a gerrit restart, but I think getting off of our fork is a good thing overall and something we haven't done for years :) | 19:25 |
| clarkb | Please carefully review that one too. | 19:25 |
| ianw | it would be good to get the latest zuul summary plugin in with better timestamps | 19:26 |
| ianw | i have | 19:26 |
| ianw | #link https://etherpad.opendev.org/p/gerrit-upgrade-2021 | 19:26 |
| ianw | tracking general upgrade notes | 19:26 |
| clarkb | ianw: landing 791995 should rebuild with the latest zuul summary plugin too. | 19:26 |
| ianw | when we have the accountReviewDb situation sorted, i think we can more or less migrate at will | 19:27 |
| clarkb | ianw: was reverse dns set up? Seeing that on your etherpad and remembering that is important for email | 19:27 |
| ianw | ahh, that's a good one, i should ping mnaser on that one | 19:27 |
| clarkb | ++ | 19:28 |
| clarkb | anything else on gerrit updates? | 19:28 |
| ianw | i think review02 will be the host now, especially with the address pinning we've done | 19:28 |
| fungi | i suppose the same mariadb container can be useful for a future zuul scheduler, and storyboard containerization as well | 19:28 |
| clarkb | fungi: yup, we also already do similar for gitea and maybe something else too | 19:29 |
| ianw | yeah, it's just the upstream container, but the deployment bits may be helpful to copy | 19:29 |
| fungi | oh, right, forgot we already had that implemented for gitea since ages | 19:29 |
| clarkb | #topic Server Upgrades | 19:30 |
| clarkb | We have already discussed the eavesdrop upgrade and tasks for the gerrit upgrade | 19:30 |
| clarkb | I'm still slowly poking at the listserv upgrades. I'm hoping I'll have time to boot a test node and upgrade it in place during the second half of this week | 19:30 |
| clarkb | I'll be using the snapshot that fungi took about a month ago t oavoid needing to wait for and slow down the server again | 19:31 |
| ianw | #link https://review.opendev.org/c/opendev/system-config/+/795207 | 19:31 |
| ianw | ^ that removes ask.o.o which is causing deployment issues as it's now shutdown | 19:31 |
| clarkb | ianw: fungi: shoudl we stash a db dump from the ask db as part of that? | 19:31 |
| ianw | we do already have that on backup hosts | 19:32 |
| clarkb | oh excellent | 19:32 |
| clarkb | the redirect with LE cert seems to be working well too | 19:32 |
| fungi | it's a local redis, right? | 19:32 |
| clarkb | no askbot is postgres or mysql. I forget which | 19:32 |
| fungi | ahh, okay, but regardless it's not in a trove instance | 19:33 |
| ianw | iirc it was a mysql and dumped out with mysqldump as a stream | 19:33 |
| clarkb | for that I am not sure | 19:33 |
| clarkb | it might be trove if it was mysql | 19:33 |
| fungi | i'll check, if it's trove i'll make an archival snapshot and then delete the instance | 19:33 |
| clarkb | thanks! | 19:33 |
| clarkb | Any other server upgrades or cleanups to call out? | 19:33 |
| ianw | we must be getting pretty thin on xenial now? | 19:34 |
| clarkb | Oh it would be good if infra-root can check the test node listservs that I upgraded to see if anything stands out as broken | 19:34 |
| clarkb | ianw: slowly but surely :) | 19:34 |
| ianw | post listserv, eavesdrop, review what is left? | 19:34 |
| ianw | paste i guess | 19:34 |
| clarkb | one thing I notice about the inplace upgrades of the listservs is we'll end up with sysv init scripts and unit files for mailman | 19:34 |
| fungi | not wiki, it's still trusty! ;) | 19:34 |
| clarkb | ianw: translate which we need to figure out because zanata, health, elasticsearch, logstash | 19:35 |
| clarkb | we said we wouldn't just turn off ELK things to give peopel time to try and figure that out. We should probably periodically check in on that and work with openstack to draw a line in the sand so that we aren't just keepign it out there forever | 19:36 |
| fungi | no trove instance for askbot, btw | 19:36 |
| clarkb | #action clarkb followup with openstack on ELK retirement planning | 19:37 |
| clarkb | I can do that | 19:37 |
| ianw | zanata i forget, was that a dead project? | 19:37 |
| fungi | (we have trove instances for one or more of cacti, grafana, paste, refstack, review, subunit2sql, translate, wiki, zuul) | 19:37 |
| clarkb | yup | 19:37 |
| fungi | one or more each i mean | 19:38 |
| clarkb | ianw: and there was talk about transitioning to a hosted open source alternative, but it kept getting deprioritized on my end because other things kept popping up | 19:38 |
| fungi | (e.g., review/review-dev/review-test) | 19:38 |
| clarkb | ianw: transitioning that would actually be a good project for someone in the community if we can find someone interested in it since all the data is exportabel as a user aiui | 19:38 |
| ianw | oh, cacti must be another one that needs updating | 19:38 |
| clarkb | corvus had talked about switching to some prometheus + influxdb + grafana type setup | 19:39 |
| ianw | yeah, i remember discussion on translate that seemed to stall | 19:39 |
| fungi | if memory serves, the proposal was to have prometheus query snmpd and then store the data in its own timeseries database? | 19:39 |
| corvus | i recommend skipping influxdb | 19:40 |
| corvus | ya | 19:40 |
| clarkb | The upside to that is we can plug services like gitea into it | 19:40 |
| corvus | grafana+prometheus snmp_exporter+grafana | 19:40 |
| ianw | i was reading last night about https://grafana.com/docs/grafana/latest/whatsnew/whats-new-in-v8-0/#grafana-v80-alerts | 19:40 |
| clarkb | and get service level as well as host level metrics | 19:40 |
| fungi | and i guess if we want history, we can use rrdtool to dump the existing graphs and ingest them into prom | 19:40 |
| corvus | i haven't run that yet; but i have run influxdb and don't think it's a good fit right now | 19:41 |
| clarkb | maybe we should write up a small spec for that? As I do think replacing cacti with prometheus gets us more functionality like the gitea service metrics which will be nice to have | 19:42 |
| clarkb | and I suspect that starting fresh might be simpler than uplifting cacti (but I'm not super familiar with either tool so could be wrong about that) | 19:43 |
| fungi | also i guess it allows us to have one grafana interface into things | 19:43 |
| fungi | so we can mix-n-match stuff from graphite and prometheus into the same dashboards | 19:43 |
| clarkb | we could include zuul system metrics alonside the service metric :) | 19:44 |
| fungi | precisely | 19:44 |
| corvus | yep. the trick will be getting actually correct memory graphs :) | 19:44 |
| clarkb | #action Someone Write up a spec to replace cacti with prometheus | 19:44 |
| clarkb | This way I won't forget its a thing | 19:44 |
| clarkb | I can probably give that a go too, but not promising that at the moment | 19:45 |
| clarkb | Any other upgrade items before we move on? | 19:45 |
| clarkb | #topic Removing PPC packages from CentOS 8 mirrors | 19:46 |
| clarkb | Yesterday we discovered that our centos mirrors had stopped updating because we ran out of afs quota room | 19:46 |
| clarkb | it was discovered that we are mirroring ppc packages btu we can't find any users of that (not surprising as we don't have ppc hardware) | 19:46 |
| fungi | apparently we intentionally included ppc64le packages in our centos-7 mirroring, and it's lumped in with 8 and 8-stream by default | 19:47 |
| ianw | did blame show me/tonyb adding it explicitly at some point? | 19:47 |
| fungi | ianw: if you did, it was many years back | 19:47 |
| clarkb | I wanted to bring this up here as a last call for objections if there is some reason to not make that change | 19:47 |
| ianw | looks like https://review.opendev.org/c/opendev/system-config/+/662727 | 19:48 |
| clarkb | otherwise I think we should update all our mirrors to drop ppc | 19:48 |
| fungi | for 8 and 8-stream we just mirror all the official architectures, which are presently x86-64, aarch64 and ppc64le | 19:48 |
| clarkb | Then update afs quotas as appropriate | 19:48 |
| ianw | hrw just posted a note not long about about dropping ppc from kolla iirc | 19:48 |
| clarkb | ya yoctozepto confirmed nothing ever used ppc kolla in our CI systems | 19:48 |
| clarkb | they had some ppc support but never exercised by us | 19:48 |
| fungi | i have a feeling we'd cut nearly a third of our content from that volume if we dropped ppc64le packages/images | 19:49 |
| clarkb | has anyone written a change to do that cleanup yet? Does anyone want to write it? | 19:49 |
| ianw | i can do that | 19:50 |
| fungi | i expect it to be a fair number of exclusions, since there are more than a few subtrees for it | 19:50 |
| *** mordred has joined #opendev-meeting | 19:50 | |
| clarkb | fungi: I think we can jsut remove ppc64le from altarchs and add it to altarches_ignored? | 19:50 |
| clarkb | ianw: thanks | 19:51 |
| fungi | for centos-7 yes, but it's not in the altarches tree in 8 and 8-stream | 19:51 |
| clarkb | fungi: aha | 19:51 |
| fungi | it's an official architecture so it's in-tree along with x86-64 and aarch64 | 19:51 |
| clarkb | #action ianw Push change to cleanup ppc packages in our CentOS mirrors | 19:51 |
| fungi | so, yeah, i welcome someone more familiar with rhel/centos repository structure tackling that | 19:52 |
| clarkb | #topic Scheduling Project Renames | 19:52 |
| clarkb | fungi: you had volunteered to update the rename playbook(s), do we have any change(s) for that yet? | 19:52 |
| fungi | i can't remember, did we determine this will be our first attempt at renaming a project since the 3.2 upgrade? | 19:52 |
| clarkb | this is our first attempt yes, other than what we did by hand on the test node to confirm it is possible | 19:53 |
| fungi | also not surprisingly i forgot i volunteered for that, will need to revisit what we determined needed to be done for it | 19:53 |
| fungi | just remove the mysql bits i suppose? | 19:53 |
| clarkb | fungi: we need to drop all the db stuff from the playbook and only do the on disk renames. Otherwise I think it is very similar | 19:53 |
| clarkb | yup | 19:53 |
| clarkb | and we can possibly test it using our gerrit functional test jobs | 19:54 |
| fungi | okay, i'll try to leave myself a less forgettable reminder this time | 19:54 |
| clarkb | by adding that playbook to the list of test playbooks and confirming a project has moved within gerrit | 19:54 |
| clarkb | but we can do that as a followon | 19:54 |
| clarkb | thanks! | 19:54 |
| clarkb | once we have those updates up then I think we can think about schedulign the renames as we'll have the tools to do it | 19:54 |
| clarkb | #topic Open Discussion | 19:54 |
| fungi | i'm happy to include the functional testing in my change, but will appreciate some guidance on the validating bits | 19:54 |
| clarkb | I can help with that | 19:55 |
| fungi | (when i reach that point) | 19:55 |
| clarkb | I've poked at similar for other changes recently | 19:55 |
| clarkb | We did end up using our time afterall. Was there anything else to bring up in the last 5 minutes here? | 19:55 |
| fungi | i have nothing | 19:55 |
| fungi | need to shift to prepping dinner in a few minutes | 19:55 |
| clarkb | Why don't we all return to eating $meal then :) | 19:56 |
| clarkb | thanks everyone! | 19:56 |
| clarkb | #endmeeting | 19:56 |
| opendevmeet | Meeting ended Tue Jun 8 19:56:47 2021 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 19:56 |
| opendevmeet | Minutes: http://eavesdrop.openstack.org/meetings/infra/2021/infra.2021-06-08-19.01.html | 19:56 |
| opendevmeet | Minutes (text): http://eavesdrop.openstack.org/meetings/infra/2021/infra.2021-06-08-19.01.txt | 19:56 |
| opendevmeet | Log: http://eavesdrop.openstack.org/meetings/infra/2021/infra.2021-06-08-19.01.log.html | 19:56 |
| fungi | thanks clarkb! | 19:57 |
| ianw | ++ | 19:57 |
| *** pleimer_ has quit IRC | 22:09 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!