| *** sboyron has joined #opendev-meeting | 07:22 | |
| *** hashar has joined #opendev-meeting | 08:11 | |
| *** tobiash has quit IRC | 08:15 | |
| *** tobiash has joined #opendev-meeting | 08:16 | |
| *** hamalq has quit IRC | 16:56 | |
| *** hamalq has joined #opendev-meeting | 16:57 | |
| clarkb | hello anyone else here for the meeting? | 19:00 |
|---|---|---|
| fungi | heyhey! | 19:00 |
| ianw | o/ | 19:00 |
| clarkb | #startmeeting infra | 19:01 |
| openstack | Meeting started Tue Nov 17 19:01:17 2020 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:01 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:01 |
| *** openstack changes topic to " (Meeting topic: infra)" | 19:01 | |
| openstack | The meeting name has been set to 'infra' | 19:01 |
| clarkb | #link http://lists.opendev.org/pipermail/service-discuss/2020-November/000135.html Our Agenda | 19:01 |
| clarkb | #topic Announcements | 19:02 |
| *** openstack changes topic to "Announcements (Meeting topic: infra)" | 19:02 | |
| clarkb | Wallaby cycle signing key has been activated https://review.opendev.org/760364 | 19:02 |
| clarkb | Please sign if you haven't yet https://docs.opendev.org/opendev/system-config/latest/signing.html | 19:02 |
| * clarkb makes a note to sit down after lunch and get this done | 19:02 | |
| ianw | i think i did | 19:02 |
| clarkb | #topic Actions from last meeting | 19:04 |
| *** openstack changes topic to "Actions from last meeting (Meeting topic: infra)" | 19:04 | |
| clarkb | #link http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-11-10-19.01.txt minutes from last meeting | 19:05 |
| clarkb | No actions recorded. Moving on | 19:05 |
| clarkb | #topic Priority Efforts | 19:05 |
| *** openstack changes topic to "Priority Efforts (Meeting topic: infra)" | 19:05 | |
| clarkb | #topic Update Config Management | 19:05 |
| *** openstack changes topic to "Update Config Management (Meeting topic: infra)" | 19:05 | |
| clarkb | ianw: looks like you've started investigating a docker deployed hound? | 19:05 |
| ianw | yes the project has had a bit of a second life and moved in github, breaking the puppet | 19:06 |
| ianw | it's going to be a pretty straight-forward container + apache, so i think it's best to just modernise it | 19:06 |
| fungi | what's still lacking for elasticsearch backend in gitea? | 19:06 |
| clarkb | fungi: a release for 1.13.0 | 19:07 |
| clarkb | then testing to see how to deploy it and all that | 19:07 |
| fungi | oh, got it. for some reason i thought that already got released | 19:07 |
| clarkb | they were down to like two issues remaining for the release on the issue tracker but it got a new rc and not a release | 19:08 |
| clarkb | maybe that means it is close? | 19:08 |
| ianw | yeah, i think it's probably best to have this in our pocket so that can be done ad-hoc, given it will be a fair bit more deployment work | 19:08 |
| clarkb | ya lot of unknowns (like sizing of elasticsearch deplioyments) | 19:09 |
| clarkb | any other config management items to bring up? | 19:09 |
| clarkb | #topic OpenDev | 19:11 |
| *** openstack changes topic to "OpenDev (Meeting topic: infra)" | 19:11 | |
| clarkb | We are just a few days from the planned Gerrit upgrade to 3.2 | 19:11 |
| clarkb | #link http://lists.opendev.org/pipermail/service-announce/2020-October/000012.html | 19:11 |
| clarkb | Need to land the non WIP changes at https://review.opendev.org/#/q/status:open+topic:gerrit-upgrade-prep | 19:12 |
| * yoctozepto is here for the meeting too | 19:12 | |
| clarkb | Looks like those have received recent reviews so after the meeting I think we can start landing them. fungi did you want to review those and maybe approve them if they look good? I will be around to help if things get sad | 19:12 |
| clarkb | https://review.opendev.org/#/c/762712/ in particular is important because it conflicts with the WIP portion of my stack so I need to rebase once this one lands | 19:13 |
| clarkb | #link https://etherpad.opendev.org/p/opendev-gerrit-3.2-upgrade-plan the plan is here. A few items still need a little work. Please take a look. | 19:13 |
| clarkb | For the items that need work the first is deciding how we want to backup the 2.13 state as part of the upgrade. After some thought I think I prefer mounting a new volume on the side, then rsyncing into it | 19:13 |
| clarkb | if you have better ideas please leave them on the etherpad | 19:14 |
| fungi | the alternative would probably be to umount the volume and then snapshot it with openstack api calls | 19:14 |
| clarkb | fungi: yup, my concern with that is that we're relying on cinder to do all that successfully for us, which it has done previously if we prefer that appraoch | 19:14 |
| fungi | down-side to that is there are extra steps involved in turning that into a mountable replacement | 19:15 |
| clarkb | The other big todo item was thinking about how to get config management into sync with the post upgrade state. I decided that squashing the 4 changes that reflect new state into one change and landing it before zuul starts is the best option. I have done that squashing and kept the old changes with distinct changeids to ease reviewing | 19:15 |
| fungi | i'm in favor of the rsync plan. i wonder if pre-syncing anything into it would speed up the final sync during the maintenance, or if all the repack/gc would still require replacing most of it | 19:16 |
| clarkb | fungi: I think we backup prior to gc'ing so maybe we should run that early | 19:16 |
| clarkb | I like that idea | 19:16 |
| clarkb | re config management syncup I need to update the etherpad to reflect that I did the squash | 19:17 |
| clarkb | If you can think of anything else to consider please reach out and update the etherpad too | 19:17 |
| clarkb | One thing to note is I tried to put time stamps/estimates in the etherpad to give people a sense for how this process should progress over time | 19:18 |
| clarkb | this info is based on what we learned doing the review-test upgrades and should be fairly accurate particularly for the gerrit upgrade steps | 19:19 |
| ianw | i agree an rsync copy seems the simplest way to keep things separate | 19:19 |
| ianw | can always flip it r/o mount to avoid any mistakes too | 19:19 |
| ianw | borg is running too :) | 19:20 |
| clarkb | yup Ifigure we've always got borg/bup to fall back on if we need it | 19:20 |
| clarkb | this is more of a convenience for quicker recovery if necessary | 19:20 |
| clarkb | cool. I'll work on updating the etherpad to better reflect those choices today | 19:23 |
| clarkb | then hopeflly the next cuple of days we can do some last minute checks as well as prep the "backup" volume | 19:23 |
| clarkb | Anything else on this topic? | 19:25 |
| ianw | just a big thanks to you and fungi for working through it! | 19:26 |
| clarkb | #topic General topics | 19:26 |
| *** openstack changes topic to "General topics (Meeting topic: infra)" | 19:26 | |
| clarkb | #topic Bup and Borg Backups | 19:26 |
| *** openstack changes topic to "Bup and Borg Backups (Meeting topic: infra)" | 19:26 | |
| clarkb | This was just mentioned and it seems like keeping up to date on progress here is a good thing too | 19:27 |
| clarkb | ianw: I know you had talked about starting the backups over now that you've trimmed them a bit. Has that happened or have we decided that isn't necessary? | 19:27 |
| ianw | yes, i did that, particularly for review and zuul | 19:27 |
| ianw | so everything has been running in production mode now for ~ a week | 19:28 |
| ianw | i think let's get through the gerrit upgrade, then we can audit the borg logs look good and retire bup | 19:28 |
| ianw | s/logs/backups | 19:28 |
| clarkb | great, figuring out when to stop using bup was my next question. Doing audits after the upgrade sounds good to me | 19:28 |
| clarkb | also we're backup up to two separte providers as a result of this as well? | 19:29 |
| clarkb | that is a good improvement regardless of the tool in use | 19:29 |
| ianw | yep, vexxhost & rax.ord with borg | 19:29 |
| ianw | the vexxhost only has a 1tb volume, as we ran out of quota. we can re-assign some of the bup quota to that i imagine | 19:29 |
| ianw | but we can also prune now, which we couldn't before with bup | 19:30 |
| clarkb | neat | 19:31 |
| clarkb | Any thing else on bup/borg/backups? | 19:31 |
| ianw | nope | 19:32 |
| clarkb | #topic General Topics | 19:32 |
| *** openstack changes topic to "General Topics (Meeting topic: infra)" | 19:32 | |
| clarkb | There were a number of other topics including, PTG followups, meetpad access in china, openstackid.org, and splitting up the puppet else job | 19:32 |
| clarkb | I've been super head down on the gerrit stuff and not had time for any of ^ myself | 19:33 |
| clarkb | is there anything new on any of these topics others would like to bring up? | 19:33 |
| clarkb | Sounds like no | 19:35 |
| ianw | none for me -- still trying to reduce puppet :) | 19:35 |
| clarkb | ++ thank you for working on that | 19:35 |
| clarkb | #topic Open Discussion | 19:35 |
| *** openstack changes topic to "Open Discussion (Meeting topic: infra)" | 19:35 | |
| clarkb | Anything else? | 19:35 |
| clarkb | Next week is a US holiday Thursday and Friday. I plan to be around for the first half of the week and will host a meeting at our regular time | 19:37 |
| clarkb | I'll likely be looking for followups to the gerrit upgrade more than anything else and try and enjoy some quiet time after a busy few months | 19:38 |
| ianw | i have visited the USA outlet store and acquired frozen cranberries for sauce, as that's not really a thing here | 19:38 |
| fungi | all you need is a bog and some cool weather... oh | 19:39 |
| clarkb | ianw: the best cranberries come in a can and look like jello | 19:41 |
| clarkb | we can probably ship some to you though I have no idea what that would cost or how long it would take | 19:41 |
| ianw | clarkb: i think that's one of those things you had to grow up with to like :) i couldn't get into the beans with mushroom soup either. but yams and marshmallows, that was a revelation | 19:42 |
| clarkb | ha | 19:42 |
| fungi | i have to disagree on the congealed canned berries. but then again i grew up with stewed fresh ones | 19:43 |
| clarkb | we only got the canned stuff because we lived on the island. Fresh and dried are relatively new to me. | 19:43 |
| fungi | anything which comes in a can should not be served still in can shape | 19:44 |
| ianw | fungi you might like to look in on https://review.opendev.org/#/c/761621/ and the follow-on https://review.opendev.org/762588 as that updates the build table to not be unselectable in zuul ui. i think i remember you having some troubles there | 19:44 |
| fungi | ianw: ooh, thanks! yes that limitation has continued to plague me | 19:44 |
| clarkb | alright sounds like that may be it. Feel free to follow up with us in #opendev or at service-discuss@lists.opendev.org | 19:45 |
| clarkb | I'll call the meeting now and we can all have 15 minutes of breakfast/lunch/dinner back :) | 19:45 |
| clarkb | thank you everyone! | 19:45 |
| clarkb | #endmeeting | 19:45 |
| *** openstack changes topic to "Incident management and meetings for the OpenDev sysadmins; normal discussions are in #opendev" | 19:45 | |
| openstack | Meeting ended Tue Nov 17 19:45:32 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 19:45 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-11-17-19.01.html | 19:45 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-11-17-19.01.txt | 19:45 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-11-17-19.01.log.html | 19:45 |
| fungi | thanks clarkb! | 19:45 |
| *** hashar has quit IRC | 21:06 | |
| *** hashar_ has joined #opendev-meeting | 21:06 | |
| *** hashar_ is now known as hashar | 21:20 | |
| *** sboyron has quit IRC | 21:58 | |
| *** hashar has quit IRC | 22:26 | |
| *** zbr has quit IRC | 23:57 | |
| *** zbr4 has joined #opendev-meeting | 23:57 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!