*** ChanServ has quit IRC | 01:05 | |
*** corvus has quit IRC | 01:10 | |
*** ChanServ has joined #opendev-meeting | 01:15 | |
*** weber.freenode.net sets mode: +o ChanServ | 01:15 | |
*** sboyron has joined #opendev-meeting | 06:17 | |
*** mordred has quit IRC | 10:12 | |
*** fungi has quit IRC | 10:13 | |
*** fungi has joined #opendev-meeting | 10:14 | |
*** mnaser has quit IRC | 10:16 | |
*** guillaumec has quit IRC | 10:18 | |
*** halali_ has quit IRC | 10:18 | |
*** tobiash has quit IRC | 10:18 | |
*** sboyron has quit IRC | 10:18 | |
*** mnaser has joined #opendev-meeting | 10:18 | |
*** sboyron has joined #opendev-meeting | 10:19 | |
*** guillaumec has joined #opendev-meeting | 10:19 | |
*** halali_ has joined #opendev-meeting | 10:19 | |
*** tobiash has joined #opendev-meeting | 10:19 | |
*** ChanServ has quit IRC | 10:21 | |
*** ChanServ has joined #opendev-meeting | 10:24 | |
*** weber.freenode.net sets mode: +o ChanServ | 10:24 | |
*** ianw has quit IRC | 10:27 | |
*** ianw has joined #opendev-meeting | 10:27 | |
*** mordred has joined #opendev-meeting | 10:36 | |
*** mordred has quit IRC | 10:41 | |
*** guillaumec has quit IRC | 11:00 | |
*** guillaumec has joined #opendev-meeting | 11:00 | |
*** mordred has joined #opendev-meeting | 12:01 | |
*** rosmaita has joined #opendev-meeting | 18:09 | |
*** mnaser has quit IRC | 18:30 | |
*** mnaser has joined #opendev-meeting | 18:31 | |
clarkb | Anyone else here for the meeting? we will get started in a few minutes | 18:58 |
---|---|---|
rosmaita | o/ | 18:59 |
fungi | howdy | 19:00 |
clarkb | #startmeeting infra | 19:01 |
openstack | Meeting started Tue May 25 19:01:25 2021 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:01 |
openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:01 |
*** openstack changes topic to " (Meeting topic: infra)" | 19:01 | |
openstack | The meeting name has been set to 'infra' | 19:01 |
clarkb | #link http://lists.opendev.org/pipermail/service-discuss/2021-May/000241.html Our Agenda | 19:01 |
ianw | o/ | 19:01 |
clarkb | #topic Announcements | 19:02 |
*** openstack changes topic to "Announcements (Meeting topic: infra)" | 19:02 | |
clarkb | I didn't have any announcements | 19:02 |
clarkb | #topic Actions from last meeting | 19:02 |
*** openstack changes topic to "Actions from last meeting (Meeting topic: infra)" | 19:02 | |
clarkb | #link http://eavesdrop.openstack.org/meetings/infra/2021/infra.2021-05-18-19.01.txt minutes from last meeting | 19:03 |
clarkb | We don't appear to have recorded any actions last meeting | 19:03 |
clarkb | #topic Priority Efforts | 19:03 |
*** openstack changes topic to "Priority Efforts (Meeting topic: infra)" | 19:03 | |
clarkb | #topic OpenDev | 19:03 |
*** openstack changes topic to "OpenDev (Meeting topic: infra)" | 19:03 | |
clarkb | I think it was fungi that mentioned it is odd we keep this as a separate meeting topic when all the things we do are basically opendev. | 19:04 |
fungi | the meeting is the opendev meeting ;) | 19:04 |
clarkb | I'll try to remember when I set up the agenda for next meeting that this needs a reorg | 19:04 |
clarkb | I don't have any updates that have historically fit under this heading that don't have agenda items later in the agenda | 19:04 |
clarkb | I think we should move on | 19:04 |
clarkb | #topic Update Config Management | 19:05 |
*** openstack changes topic to "Update Config Management (Meeting topic: infra)" | 19:05 | |
clarkb | This one is in a similar situation. We tend to talk about these updates in the context of saying updating mailman config management | 19:05 |
clarkb | I'm thinking maybe instead of putting "priority efforts" upfront we just try to order the agenda item lists to ensure the most important items come first | 19:05 |
fungi | wfm | 19:05 |
clarkb | (some weeks that will be our priority efforts and others it will be out of the blue important things) | 19:05 |
ianw | ++ | 19:06 |
fungi | it was originally a section for covering activities related to specs marked as priority | 19:06 |
clarkb | cool, with that I think we can jump into the agenda items which I've already applied a priority sorting function to | 19:06 |
clarkb | #topic General topics | 19:07 |
*** openstack changes topic to "General topics (Meeting topic: infra)" | 19:07 | |
clarkb | #topic Refreshing non LE certs | 19:07 |
*** openstack changes topic to "Refreshing non LE certs (Meeting topic: infra)" | 19:07 | |
clarkb | #link https://review.opendev.org/c/opendev/system-config/+/792789 Redirect ask.o.o | 19:07 |
clarkb | ianw started work on making a landing page for a shtudown ask.openstack.org. This change lgtm though dns for acme challenges needs updating | 19:07 |
*** sboyron has quit IRC | 19:07 | |
clarkb | That switches ask.o.o to an LE cert then serves a tiny index.html pointing people to the internet archive | 19:08 |
clarkb | Separately I've got a stack of changes to convert ethercalc, storyboard, and translate to LE certs | 19:08 |
clarkb | #link https://review.opendev.org/c/opendev/system-config/+/792708 stack of LE certs | 19:08 |
ianw | thanks, yeah i can finalise that | 19:08 |
clarkb | These changes will handle the bulk of the expiring certs. | 19:09 |
clarkb | We are left with wiki, openstackid and openstackid-dev | 19:09 |
clarkb | fungi: for wiki is the easiest thing to buy a cert? | 19:09 |
clarkb | or is it under sufficient config management to use LE? | 19:09 |
fungi | yes probably | 19:09 |
fungi | to buy a cert | 19:09 |
fungi | it is under absolutely zero config management | 19:10 |
clarkb | ok, in the past I've done that about a week before expiry. I can work that out when we get closer to expiration | 19:10 |
clarkb | For openstackid and openstackid-dev fungi and I spoke to the foundation sysadmins and they are willing to take on management of those services. However, it is unlikely that will be done before the certs expire | 19:10 |
fungi | the config management we have for mediawiki is mostly sufficient to stand up a working server, but the data needs reorganizing compared to how it's laid out on the production server | 19:11 |
clarkb | I told them I would refresh the cert for the prod server one way or another. My plan is to try and LE it if the others I am LEing go well (they should be fine) | 19:11 |
*** corvus has joined #opendev-meeting | 19:11 | |
fungi | also "mostly" is important there, i did not get so far as to work out why openid auth was broken on the dev deployment | 19:11 |
clarkb | fungi: gotcha | 19:11 |
fungi | (for wiki) | 19:11 |
clarkb | for openstackid-dev it sounded like we don't think anything needs a valid cert for that anymore so I'm going to let it just expire | 19:11 |
clarkb | if we discover after it expires that something does need it we can provision it with LE too (I wanted to avoid provisioning a cert if someone else is going to manage it with their own config management later) | 19:12 |
clarkb | Long story short we have a plan now and are making progress, we should have things orted well before any certs expire | 19:13 |
clarkb | #topic Potentially Migrating away from Freenode to OFTC | 19:13 |
*** openstack changes topic to "Potentially Migrating away from Freenode to OFTC (Meeting topic: infra)" | 19:13 | |
clarkb | Last week freenodes staff quit due to changes to the organization. Since then there have been changes to things like use policies for freenode too | 19:13 |
clarkb | this is precipitated conversations around whether or not we should stick to freenode | 19:14 |
clarkb | for many yaers we have maintained a small backup presence on oftc | 19:15 |
clarkb | increasing it appears that switching to oftc is going to be a good moev for us, but we are soliciting feedback from our membr projects | 19:15 |
clarkb | #link http://lists.opendev.org/pipermail/service-discuss/2021-May/000236.html Fungi's call for input | 19:15 |
clarkb | Kata and Zuul have both responded that they are happy to move (zuul would like to stick close to ansible if possible but that isn't critical) | 19:16 |
clarkb | openstack seems to be dragging its feet | 19:16 |
clarkb | fungi: have we heard anything from airship or starlingx? | 19:16 |
fungi | well, rather the tc is trying to crowdsource input on an irc preference from all the openstack projects | 19:16 |
fungi | i've heard nothing from either airship or starlingx leadership | 19:17 |
fungi | nor openinfralabs, whose ml i also contacted | 19:17 |
clarkb | fungi: maybe we should ask ildikov and ashlee to prod them? | 19:17 |
clarkb | and I guess have you prod openinfralabs again? (I think you interface wit hthem) | 19:18 |
ianw | https://github.com/ansible-community/community-topics/issues/19 appears to still be ansible discussing things? | 19:18 |
fungi | can't hurt, though for the most part i'm taking a lack of response to mean they're fine with whatever we decide to go with | 19:18 |
clarkb | ianw: ya from corvus' email it seemed ansible was undecided | 19:18 |
corvus | that's my understanding | 19:18 |
fungi | ianw: yeah, also the python community, the freebsd community and some other large ones are still trying to decide what to do | 19:18 |
corvus | i feel like these are all feathers that tip the scale really; i don't think there are many folks that feel very strongly oftc vs libera. | 19:19 |
corvus | there are folks that feel strongly "not freenode" | 19:19 |
clarkb | the recent updates to freenode policies are concerning so ya "not freenode" seems to be picking up more steam | 19:20 |
clarkb | fungi: it would probably be good to keep the pressure on so that we can make a decision sooner than later. As stalling puts us in a weird spot. I can reach out to ildikov and ashlee to see what they say about airship and starylingx | 19:23 |
clarkb | (I cannot type today, or most dats) | 19:23 |
fungi | yeah, i agree. i mostly expect openstack to take the longest to come to an answer, given larger ships take longer to turn | 19:24 |
clarkb | cool, anything else to add to this? maybe you want to talk about your bot modifications? | 19:24 |
corvus | based on what i've read, we may expect problems from freenode if we vacate. that's worth planning for. | 19:24 |
clarkb | corvus: specifically around repurposing channels right? | 19:24 |
corvus | yeah. i expect our idea of closing channels with forwarding messages is incompatible with how the new staff sees things. | 19:25 |
clarkb | ya I guess that means if we switch we need to overcommunicate so that people can discover the new location without relying on messages in irc | 19:26 |
corvus | yeah. i think that would be prudent. if nothing bad comes to pass, there's no downside. :) | 19:26 |
ianw | i've seen one incident with something to do with a hacker-news bot repeater channel | 19:26 |
ianw | but i now have several moved channels | 19:27 |
ianw | #cryptography-dev went mute and changed topic, centos seems to now be #centos-has-moved | 19:28 |
clarkb | ya I don't expect much trouble, but being prepared for it can't hurt | 19:29 |
corvus | i suspect it has more to do with areas of personal interest | 19:30 |
fungi | right, #openbsd has had a forwarding message in its topic for nearly a week already | 19:30 |
corvus | so we're probably fine. but maybe don't rule it out. | 19:30 |
clarkb | ++ | 19:31 |
fungi | i agree, it's more likely to come up when moves get press, or are otherwise high-profile and present marketing opportunities or damage control | 19:31 |
corvus | slightly OT: i'm engaging in a matrix experiment for learnings. | 19:32 |
corvus | don't need to discuss now, but just wanted to share | 19:32 |
ianw | ++ experimenting with matrix > talking about matrix :) | 19:33 |
clarkb | cool. Anything else on this topic or should we move on? | 19:33 |
fungi | sean-k-mooney and yoctozepto have been as well | 19:33 |
clarkb | fungi: ^ feel free to discuss your updates or whatever other relevant items are in play | 19:33 |
fungi | wip changes are up with an "oftc" review topic | 19:33 |
fungi | i'm currently wrestling the state machine in accessbot to get it working | 19:34 |
clarkb | #link https://review.opendev.org/q/topic:oftc Bot updates to make oftc a possibility | 19:34 |
fungi | the biggest difference there is that the acls on oftc use coarse-grained access levels rather than individual access flags | 19:34 |
fungi | so our mask handling has to be ripped out and replaced | 19:34 |
corvus | fungi: are you sure it has to be ripped out? i *thought* i left some provision for swapping in roles instead of flags | 19:35 |
fungi | also the lack of identify-msg cap on their ircd means the initial connection stages are subtly different, but enough i'm having trouble getting the identify message to fire at the right time | 19:35 |
corvus | (but it's entirely possible that it grew more complex since then and overwhelmed whatever provision i thought i left for that) | 19:36 |
fungi | corvus: if it can also support rbac then that'd be a huge help, but it wasn't especially obvious if it's in there | 19:36 |
corvus | where is the code now? :) | 19:36 |
fungi | in my editor, i'll push up a broken i progress change | 19:37 |
fungi | er, in progress change | 19:37 |
corvus | ah looks like ./docker/accessbot/accessbot.py ? | 19:37 |
fungi | yes, that's the current state | 19:37 |
fungi | i thought you were asking if my work in progres on that script was up for review yet, sorry | 19:37 |
corvus | fungi: i think you're right and i'm wrong. i think it needs to be ripped out and replaced. | 19:38 |
fungi | cool, i'm about halfway into that, at least | 19:38 |
fungi | thanks for looking | 19:38 |
corvus | sorry for the red herring | 19:38 |
fungi | no worries, that code has a thick layer of dust on it | 19:38 |
clarkb | We have a few more topisc to get through, lets get through those and can swing back to this if we still have time | 19:39 |
fungi | anyway, that's all i've got. going to keep plugging on the accessbot work | 19:40 |
fungi | thanks! | 19:40 |
clarkb | #topic Switch Vexxhost to provide only specialized labels in Nodepool | 19:40 |
*** openstack changes topic to "Switch Vexxhost to provide only specialized labels in Nodepool (Meeting topic: infra)" | 19:40 | |
clarkb | fungi: I didn't manage to find this change. But the tl;dr is that the difference in base memory for vexxhost nodes make its problematic for landing things that need more memory then later run elsewhere with less memory | 19:40 |
clarkb | considering the size of vexxhost's pool slice we can (with reasonable safety) shift vexxhost to only providing the specialized resources and not the generic ones | 19:41 |
* clarkb looks for the change again | 19:41 | |
*** halali_ has quit IRC | 19:41 | |
clarkb | #link https://review.opendev.org/c/openstack/project-config/+/785769 | 19:41 |
fungi | #link yes, sorry, i was pulling up the ml thread | 19:41 |
clarkb | fungi: looks like it needs to be reabsed. Is there any reason to keep it as WIP? | 19:41 |
fungi | #link http://lists.opendev.org/pipermail/service-discuss/2021-April/000219.html Recent nodepool label changes | 19:42 |
fungi | i didn't want it to merge until it had been discussed, but the ml thread just sort of petered out | 19:42 |
clarkb | gotcha | 19:42 |
fungi | and yeah, i'm not surprised it's fallen into a merge conflict in the meantime | 19:43 |
clarkb | do you want us to try and pick up the discussion on the mailing list? | 19:43 |
clarkb | fwiw I'm happy to make that change (and could say so on the mailing list) | 19:43 |
ianw | do we have a sense of how many jobs use those expanded node types? | 19:43 |
clarkb | basically try and be more flexible in allocating resources to both avoid problems and maybe provide more interesting flavors where they can be useful | 19:43 |
clarkb | ianw: I think it has fluctuated over time. octavia uses the nested virt flavors pretty consistently, but then sean-k-mooney does things with the bigger and nested virt flavors when nova has specific needs iirc | 19:44 |
clarkb | airship is also a large flavor user for their main end to end test job | 19:44 |
*** halali_ has joined #opendev-meeting | 19:44 | |
fungi | i switched it out of wip | 19:45 |
fungi | and will rebase to clear the merge conflict | 19:45 |
ianw | we could only set mem= type kernel command lines to limit the ram by setting it in a dib element, right? | 19:45 |
clarkb | ianw: we can but if we do that we need different images | 19:46 |
clarkb | because we want to use more memory in other circumstances | 19:46 |
ianw | right, yeah. we used to do that iirc | 19:46 |
clarkb | ya we did it for hpcloud when there were only two clouds and the images were different anyway | 19:47 |
clarkb | its probably still worth thinking through to see if there is a way to make that happen (like can nova set boot params for us? or have a pre run playbook update the node and reboot if necessary if the time loss isn't too bad) | 19:47 |
clarkb | but fungi's change is the simplest thing we can land right now | 19:47 |
fungi | second simplest | 19:48 |
fungi | but less wasteful than just removing that region | 19:48 |
ianw | ++. i wonder if we can hotplug remove memory | 19:48 |
fungi | plz_to_be_removing_sticks_of_ram | 19:48 |
clarkb | I think we can proceed with fungi's change then revert if we find an alternative | 19:49 |
clarkb | #topic Server Upgrades | 19:49 |
*** openstack changes topic to "Server Upgrades (Meeting topic: infra)" | 19:49 | |
clarkb | (running out of time so want to keep going) | 19:49 |
clarkb | The mailman ansible stuff seems to have gone well. I don't know that we haev created any new lists yet (but this is tested with zuul) though | 19:50 |
*** halali_ has quit IRC | 19:50 | |
clarkb | Next up is taking a snapshot of the server and upgrading it through to focal. One thought I had was before we bother with a snapshot of the server I can test this on a zuul node that we hold | 19:50 |
clarkb | once we have this working reliably on the held zuul nodes (can rerun ansible against the focal node too etc) then take an up to date snapshot and make sure it runs happily on our snapshot? | 19:51 |
clarkb | do we think running through it on the snapshot is useful if doing it on a zuul node? (how careful do we want to be is the question I guess) | 19:51 |
ianw | ++ .. having gone through quite a few in-place updates with afs i didn't really have any issues | 19:51 |
ianw | the only problem was a stuck volume on the afs sever which was barely related | 19:52 |
clarkb | I'll try to make time for the held zuul node upgrade late this week or next week and we can plan doing the actual snapshot copy after that if we feel it is necessary | 19:52 |
fungi | i feel like having the snapshot on hand as a fallback is probably fine, probably the only actual thing you won't be testing with the in-place zuul upgrade is the newer mailman packages | 19:53 |
clarkb | fungi: ya and last time we had a few issues related to those iirc | 19:53 |
clarkb | had to do with new pipeline entries for mailman iirc | 19:53 |
clarkb | we worked through them then, we can probably work through it again | 19:53 |
clarkb | another consideration is we may want to avoid mailing list outages while IRC is still somewhat up for discussion | 19:54 |
clarkb | but that is a problem to worry about when we are ready to upgrade :) | 19:54 |
clarkb | ianw: any gerrit upgrade updates? | 19:55 |
ianw | no but i swear i'm getting back to it | 19:55 |
clarkb | ok :) we have all been busy | 19:55 |
clarkb | #topic Scheduling project renames | 19:55 |
*** openstack changes topic to "Scheduling project renames (Meeting topic: infra)" | 19:55 | |
clarkb | related to being busy I feel like we're not in a great spot to try and plan for this right now with everything else going on | 19:55 |
clarkb | If we can dig out of the backlog and get the playbooks updated we can start scheduling properly | 19:56 |
clarkb | #topic Open Discussion | 19:56 |
*** openstack changes topic to "Open Discussion (Meeting topic: infra)" | 19:56 | |
clarkb | rosmaita: did you have updates or more to discuss about zuul comments in gerrit? | 19:56 |
clarkb | (wanted to make sure I saved some time for the catch all topic if so) | 19:56 |
rosmaita | no, i did have a different topic though | 19:57 |
rosmaita | somehow got nuked off the agenda :) | 19:57 |
fungi | we did have someone come into the opendev channel asking how to set tags on gerrit comments, probably related to running a cinder third-party ci system | 19:57 |
clarkb | rosmaita: I may have accidentally cleaned up that topic and the old one thinking it was just the old one | 19:57 |
clarkb | rosmaita: apologies if that is what happened | 19:57 |
clarkb | rosmaita: go for it though | 19:57 |
ianw | (happy to run slightly over for missed topics) | 19:58 |
rosmaita | thanks | 19:58 |
rosmaita | sorry, my system went unstable for a minute there | 19:59 |
rosmaita | quick question about publishing the api-ref | 19:59 |
rosmaita | https://docs.openstack.org/api-ref/block-storage/index.html | 19:59 |
rosmaita | that has both v2 and v3 | 19:59 |
rosmaita | we are removing v2 this cycle, so ordinarily would also remove the v2 api-ref | 19:59 |
rosmaita | but, it could be useful for people who have procrastinated | 20:00 |
rosmaita | the api-ref isn't branched like the regular docs are | 20:00 |
fungi | right, because the apis aren't versioned the same as the software | 20:00 |
rosmaita | so i was wondering if there is a way to have the v2 api-ref built from the stable/wallaby branch or something like that? | 20:00 |
clarkb | rosmaita: can you change https://docs.openstack.org/api-ref/block-storage/v2/index.html to say (Removed in Xena) instead of (DEPRECATED) and maybe put a warning there but keep the info there? | 20:01 |
rosmaita | or is the best think just to say "REMOVED" in big letters at the top of the v2 ref | 20:01 |
rosmaita | clarkb: yeah, i guess that's the best thing | 20:01 |
clarkb | I think the reason the api docs are separated from the projects is that the apis live longer than the master branch | 20:01 |
fungi | well, it's removed from the latest version of the software, but not retroactively removed from earlier versions of the software, and that document is in theory about all openstack cinder versions | 20:02 |
clarkb | (thats probably not hte most accurate statement, but iirc the info that generates those docs isn't directly tied to the code for this reason) | 20:02 |
clarkb | fungi: ++ | 20:02 |
rosmaita | ok, a note makes sense | 20:02 |
rosmaita | for glance we put up a note telling users to generate the v1 api-ref for themselves if they really wanted it | 20:02 |
fungi | rosmaita: we're probably well off-topic for opendev, but consider it like the sdk. should the latest version of the sdk rip out cinder v2 api support and just declare old cloud deployments are no longer valuable? | 20:02 |
rosmaita | well, we keep the documentation branched with the software | 20:03 |
rosmaita | and you can use an older version of the sdk if you need to | 20:04 |
rosmaita | anyway, doesn't matter | 20:04 |
fungi | sort of. it's in the same repository as the software these days (it wasn't always) but in theory only the master branch version is maintained since that's the only one which gets published | 20:04 |
fungi | also you can't use old versions of the sdk if you have one piece of software which needs to talk to two clouds which have different supported api versions | 20:04 |
fungi | (there are people who interface with more than one cloud at the same time) | 20:05 |
rosmaita | ok, we'll put a note on the v2 ref, that will work | 20:05 |
clarkb | cool, glad we could help | 20:05 |
clarkb | Anything else? we are over time so I'll end us here if not | 20:05 |
fungi | ultimately, it's more of a question for the openstack tc and tech writing sig though | 20:05 |
rosmaita | that's all from me, other than a thank you for being ready for the freenode -> OFTC change | 20:06 |
fungi | i don't have anything else | 20:06 |
clarkb | thanks everyone! | 20:06 |
clarkb | #endmeeting | 20:06 |
*** openstack changes topic to "Incident management and meetings for the OpenDev sysadmins; normal discussions are in #opendev" | 20:06 | |
openstack | Meeting ended Tue May 25 20:06:23 2021 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 20:06 |
openstack | Minutes: http://eavesdrop.openstack.org/meetings/infra/2021/infra.2021-05-25-19.01.html | 20:06 |
openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/infra/2021/infra.2021-05-25-19.01.txt | 20:06 |
openstack | Log: http://eavesdrop.openstack.org/meetings/infra/2021/infra.2021-05-25-19.01.log.html | 20:06 |
fungi | thanks clarkb! | 20:06 |
*** rosmaita has left #opendev-meeting | 20:29 | |
*** corvus has quit IRC | 21:51 | |
*** corvus has joined #opendev-meeting | 22:04 | |
*** tobiash has quit IRC | 22:24 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!