| *** hamalq has quit IRC | 00:55 | |
| *** hamalq has joined #opendev-meeting | 03:29 | |
| *** hamalq_ has joined #opendev-meeting | 03:31 | |
| *** hamalq has quit IRC | 03:35 | |
| *** hamalq_ has quit IRC | 03:53 | |
| *** hamalq has joined #opendev-meeting | 05:42 | |
| *** hamalq has quit IRC | 05:46 | |
| *** hamalq has joined #opendev-meeting | 06:02 | |
| *** hamalq has quit IRC | 06:07 | |
| *** hamalq has joined #opendev-meeting | 06:15 | |
| *** hamalq has quit IRC | 06:20 | |
| *** hamalq has joined #opendev-meeting | 06:49 | |
| *** hamalq has quit IRC | 06:53 | |
| *** hamalq has joined #opendev-meeting | 07:05 | |
| *** hamalq has quit IRC | 07:09 | |
| *** hamalq has joined #opendev-meeting | 15:46 | |
| *** hamalq has quit IRC | 16:31 | |
| *** hamalq has joined #opendev-meeting | 16:33 | |
| *** hamalq_ has joined #opendev-meeting | 16:35 | |
| *** hamalq has quit IRC | 16:38 | |
| clarkb | anyone else here for the meeting? we'll get started shortly | 19:00 |
|---|---|---|
| fungi | sounds like a fun time | 19:00 |
| ianw | o/ | 19:00 |
| fungi | sure, why not | 19:00 |
| clarkb | #startmeeting infra | 19:01 |
| openstack | Meeting started Tue Jun 9 19:01:04 2020 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:01 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:01 |
| *** openstack changes topic to " (Meeting topic: infra)" | 19:01 | |
| openstack | The meeting name has been set to 'infra' | 19:01 |
| clarkb | #link http://lists.opendev.org/pipermail/service-discuss/2020-June/000034.html Our Agenda | 19:01 |
| clarkb | #topic Announcements | 19:01 |
| *** openstack changes topic to "Announcements (Meeting topic: infra)" | 19:01 | |
| clarkb | No announcements were listed | 19:01 |
| mordred | o/ | 19:02 |
| clarkb | #topic Actions from last meeting | 19:02 |
| *** openstack changes topic to "Actions from last meeting (Meeting topic: infra)" | 19:02 | |
| clarkb | #link http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-06-02-19.03.txt minutes from last meeting | 19:03 |
| clarkb | Last week's meeting was informal and we ended up debugging the meetpad/jitsimeet/etherpad/xmpp case sensitivity thing | 19:03 |
| clarkb | No explicit actions came out of that that we recorded. But I think it gave us a better understanding of what we can do to make that case handling difference less confusing | 19:04 |
| corvus | o/ | 19:04 |
| fungi | seems like we have a plan for it though | 19:04 |
| fungi | or at least some consensus of things we can do | 19:04 |
| clarkb | ya I think what we've found is that case confusion is a thing and we should probably switch to enforcing lower case in etherpad to avoid that anyway | 19:04 |
| clarkb | then we've got to deal with renaming/merging pads as necessary to handle that | 19:05 |
| clarkb | #topic Specs approval | 19:06 |
| *** openstack changes topic to "Specs approval (Meeting topic: infra)" | 19:06 | |
| clarkb | This spec isn't ready for approval yet, but I wanted to call it out | 19:06 |
| clarkb | #link https://review.opendev.org/#/c/731838/ Central Authentication Service spec | 19:06 |
| fungi | yeah, it needs some heavy editing | 19:06 |
| clarkb | fungi: I think we half expect a new PS based on conversation we had at the PTG? | 19:06 |
| fungi | good feedback in there from neal too | 19:06 |
| fungi | yes, you can half expect it, but i fully intend to provide it ;_ | 19:07 |
| fungi | just might not come this week | 19:07 |
| fungi | we'll see | 19:07 |
| clarkb | thanks | 19:08 |
| clarkb | #topic Priority Efforts | 19:08 |
| *** openstack changes topic to "Priority Efforts (Meeting topic: infra)" | 19:08 | |
| clarkb | #topic Update Config Management | 19:08 |
| *** openstack changes topic to "Update Config Management (Meeting topic: infra)" | 19:08 | |
| clarkb | The main topic I wanted to bring up here was the reorganization of our ansible inventory, groups, *vars, and base playbook | 19:08 |
| clarkb | What we've realized is that the vast majority of the base playbook is not service specific. It configures admin users and exim for email and ntp and so on. | 19:09 |
| clarkb | But the playbook runs against all hosts which means if any one of them fails then playbook fails. This can then cause problems if you wanted letsencrypt to run on a specific host or zuul to be updated and those hosts were fine | 19:10 |
| clarkb | in order to make that more reliable we've split the iptablse role out of base as it is service specific and put that into our service roles. Then we can decouple running base as a requirement before every service update | 19:10 |
| clarkb | mordred: ^ is that a reasonable summary of the change? Anything else to add to that? | 19:10 |
| mordred | I think that's great | 19:10 |
| clarkb | from the operator side of things be aware files haev moved around and some config has been updated. You may need to rebase outstanding changes in system-config | 19:11 |
| clarkb | Any other configuration management items to bring up? | 19:12 |
| mordred | I think that's about it - we may have discovered we're actually ok to run zuul-executor in containers | 19:13 |
| mordred | corvus is goign to verify - but I think I found that to be true now on friday | 19:13 |
| mordred | so I've got some patches up to do that | 19:13 |
| clarkb | mordred: the thought there is we have to give the container some additional permissions? | 19:13 |
| mordred | clarkb: turns out we don't seem to need anything past privileged | 19:14 |
| corvus | locally i think i saw it working in bwrap but behaving weirdly inside docker itself. but it sounds like mordred saw something different when trying on ze01 | 19:14 |
| mordred | yeah | 19:14 |
| mordred | so it's possible there are differences wrt kernel versio or docker version from the original test - or who knows | 19:14 |
| mordred | but i did bwrap inside of docker and it SEEMED to do the right things | 19:14 |
| corvus | based on what i saw, i think we should be "okay" to do it without the seccomp stuff, but i think it might be more comfortable with seccomp | 19:15 |
| corvus | mordred: did you test out afs inside docker but not in bwrap? | 19:15 |
| mordred | corvus: I think so? | 19:15 |
| corvus | k | 19:15 |
| mordred | corvus: but - let's double-check :) | 19:15 |
| corvus | so if what mordred saw holds, then i agree, we should be gtg without anything else | 19:15 |
| corvus | i'll do this after the meeting | 19:16 |
| corvus | ^ = confirm mordred's tests | 19:16 |
| mordred | if that works - we'll just be down to nodepool builder on arm running non-containerized - and we need to swing back around to that issue anyway | 19:16 |
| clarkb | the arm nodepool builder is hung up on the odd stream crossing we saw with multi arch docker builds right? | 19:16 |
| mordred | yeah - which we need to reproduce and figure out what's going on | 19:17 |
| ianw | i can probably make some tiem for at least reproduction | 19:17 |
| clarkb | #topic OpenDev | 19:19 |
| *** openstack changes topic to "OpenDev (Meeting topic: infra)" | 19:19 | |
| clarkb | #link http://lists.opendev.org/pipermail/service-discuss/2020-May/000026.html Advisory Board thread. | 19:19 |
| clarkb | The advisory board "recruiting" is still in progress. At the PTG we discussed that a gentle reminder to those who haven't responded is a good idea and then we'll move forward in a few weeks with who we get. | 19:19 |
| clarkb | The thought is that by having some involvement we can generate interest and an example of what the system is there for | 19:20 |
| clarkb | I plan to send out those gentle reminders today | 19:20 |
| fungi | like a snowball rolling downhill | 19:20 |
| corvus | in june? | 19:21 |
| clarkb | On the service disde of things Gitea 1.12.0 has had its second rc tag and I've got a change up to test a deployment of that. Looks like they've already added some additional bug fixes on top of that. We should hold off until the actual release I expect | 19:21 |
| clarkb | corvus: in some parts of the world | 19:21 |
| fungi | corvus: feel like taking a trip to chile? ;) | 19:21 |
| corvus | fungi: yes | 19:21 |
| clarkb | the good news is the templates have been very stable between rc1 and rc2 so any final release should be really close to ready and its just a matter of updating the tag I hope | 19:21 |
| fungi | i've also got a change up for upgradnig the version of etherpad. supposedly a major cause of the "broken" pads is addressed with it | 19:22 |
| clarkb | I'm excited for this update as it adds caching of git commit info which should drastically speed up our rendering of repos with large histories like nova | 19:22 |
| fungi | now that the ptg is done, this may be a good time for etherpad upgrades again | 19:22 |
| clarkb | fungi: ++ I think we can land and deploy that as soon as we are happy with the change and its testing | 19:23 |
| fungi | just double-checked and 1.8.4 is still the latest release | 19:23 |
| corvus | what does "broken" mean? | 19:24 |
| clarkb | corvus: i think like the clarkb-test etherpad on the old etherpad-dev server | 19:25 |
| clarkb | corvus: etherpads that eventually stop serving correctly | 19:25 |
| fungi | yeah, the ones which hang with "loading..." | 19:25 |
| corvus | ack | 19:26 |
| clarkb | Anything else on OpenDev or shoudl we moev on? | 19:26 |
| clarkb | (I can't type today) | 19:26 |
| fungi | i mentioned the change some weeks back in #opendev, but when we hit one of those there are telltale errors in the log which are referenced by the fix | 19:26 |
| fungi | so fingers crossed anyway | 19:26 |
| clarkb | #topic General Topics | 19:27 |
| *** openstack changes topic to "General Topics (Meeting topic: infra)" | 19:27 | |
| clarkb | #topic Project Renames | 19:28 |
| *** openstack changes topic to "Project Renames (Meeting topic: infra)" | 19:28 | |
| clarkb | I want to start with this one to make sure we get a chance to talk about it | 19:28 |
| clarkb | we had pencilled in June 12 which is this Friday. Unfortunately I've discovered I have a kids doctor visit at ~1800UTC that day | 19:28 |
| clarkb | I'm happy to go ahead with it and help as I can (we can do it early friday or later friday and I'll be around) or move it to another day if we don't have enough people around | 19:29 |
| clarkb | also we've added a few more renames since we last talked about this, the openstack foundation interop repos are getting moved now I guess | 19:29 |
| fungi | also it sounds like the openstack tc may want to rename a few more repos out of the openstack namespace into the osf namespace (relating to osf board of directors committees/working groups) | 19:29 |
| fungi | er, yeah what you just said | 19:30 |
| clarkb | fungi: yup gmann added that to the list of things about half na hour agao | 19:30 |
| fungi | perfect | 19:30 |
| clarkb | do we have any volunteers for Friday other than myself? | 19:30 |
| fungi | i'll be around | 19:30 |
| fungi | happy to do renames | 19:31 |
| clarkb | fungi: cool do you have a perference on time and I'll do my best to be around to help ? | 19:31 |
| fungi | let's say not 18:00 utc in that case... | 19:31 |
| clarkb | I can start as early as 1400UTC, then have cut off around 1730UTC, and expect to be back around 2030 UTC | 19:31 |
| clarkb | (it'll likely be shorter than that but you never know with those visits) | 19:31 |
| corvus | i should be around but would like not to drive | 19:32 |
| fungi | my schedule is wide open friday. are there other volunteers with time constraints? i could certainly accommodate either of those windows | 19:32 |
| fungi | 21:00 would work for me if that helps others | 19:32 |
| clarkb | That works for me and should give me plenty of padding on my schedule | 19:33 |
| clarkb | why don't we go with that then. Thank you fungi ! | 19:34 |
| fungi | let's do that then, we can always do some prep earlier in the day in anticipation too | 19:34 |
| clarkb | ++ thanks | 19:34 |
| clarkb | Between now and then we'll want to construct the yaml input to the renaming process and commit it to opendev/project-config once the renames happen | 19:34 |
| fungi | yep | 19:34 |
| clarkb | I can help coordinate with you to make sure we are ready by Friday | 19:35 |
| fungi | sounds good, thanks | 19:35 |
| clarkb | #topic Pip and Virtualenv Next Steps | 19:35 |
| *** openstack changes topic to "Pip and Virtualenv Next Steps (Meeting topic: infra)" | 19:35 | |
| clarkb | ianw: ^ Any update on this subject? | 19:35 |
| clarkb | I believe I saw at least one project (octavia) testing that the chagnes don't break them which was reassuring | 19:35 |
| ianw | yeah, i didn't get any complaints, and some people saying things worked | 19:36 |
| ianw | #link https://review.opendev.org/734428 | 19:36 |
| ianw | that's the review to drop it, so ... i guess we just do it? i'm not sure what else to do | 19:36 |
| fungi | wfm | 19:37 |
| clarkb | we've communicated it, at least some people have done testing and reinforced the expectation that this will be low impact, I think the next step is to land the change | 19:37 |
| AJaeger | ++ | 19:38 |
| fungi | this is also early enough in openstack's release cycle that any resulting disruption can be addressed at a comfortable pace | 19:38 |
| ianw | the one to watch for is if people say virtualenv is missing | 19:39 |
| ianw | their best bet is to add "ensure-virtualenv" role | 19:39 |
| AJaeger | ianw: please send an email once we merge the change | 19:39 |
| clarkb | a followup to the announcement thread would be good indicated we've landed the change once that happens | 19:39 |
| clarkb | AJaeger: ++ | 19:39 |
| ianw | will do | 19:39 |
| clarkb | anything else on this topic? | 19:40 |
| ianw | no, thanks | 19:40 |
| clarkb | #topic DNS Cleanup | 19:41 |
| *** openstack changes topic to "DNS Cleanup (Meeting topic: infra)" | 19:41 | |
| clarkb | ianw: did we end up publishing the contents for comment yet? | 19:41 |
| ianw | it looks like the backup went into merge failure | 19:41 |
| ianw | #link https://review.opendev.org/#/c/728739/ | 19:41 |
| ianw | but it would be good to merge that | 19:42 |
| ianw | the one to look through is | 19:42 |
| ianw | #link https://etherpad.opendev.org/p/rax-dns-openstack-org | 19:42 |
| ianw | perhaps to make it more manageable, if people want to delete from that things that should definitely stay, it will reduce it | 19:43 |
| clarkb | thanks and I guess we can just mark that up with comments around what can be removed? | 19:43 |
| clarkb | ah ya I see the note about removing things that should definitely stay, thanks | 19:43 |
| clarkb | I'll try to take a look at that today | 19:44 |
| clarkb | #topic PTG Recap | 19:45 |
| *** openstack changes topic to "PTG Recap (Meeting topic: infra)" | 19:45 | |
| clarkb | #link http://lists.opendev.org/pipermail/service-discuss/2020-June/000035.html Recap Email | 19:45 |
| clarkb | I wrote a long email trying to cover the important bits of the PTG for us | 19:45 |
| clarkb | Overall I think it went well. | 19:45 |
| clarkb | From an operations side meetpad seemed to work with most of its scaling issues being client side | 19:46 |
| clarkb | there were some annoying things like the etherpad focus going away when people talked sometimes and needign to reconnect because all sound went away | 19:46 |
| clarkb | but overall it held up and the groups using it seemed happy (though groups with more than 20 had less success) | 19:46 |
| clarkb | As participants we managed to get through our agenda. I think the total of 6 hours was about correct for us | 19:47 |
| clarkb | #link https://etherpad.opendev.org/p/June2020-PTG-Feedback Provide your PTG event feedback | 19:47 |
| fungi | i was pleased with the way it worked out | 19:47 |
| clarkb | the PTG organizers are soliciting feedback on the etherpad I just linked. Feel free to add your thoughts there | 19:47 |
| corvus | i have heard from folks they'd like to continue (trying) to use meetpad in the future; i think we can/should wind down pbx in favor of meetpad | 19:47 |
| clarkb | corvus: ++ | 19:48 |
| fungi | i concur | 19:48 |
| clarkb | One of the things we talked about was getting off of pytho3n for our little tools and utilities as well as services. | 19:48 |
| fungi | we lose the dial-in trunk though | 19:48 |
| clarkb | I've started to try and put together an audit of the todo list around that | 19:48 |
| clarkb | #link https://etherpad.opendev.org/p/opendev-tools-still-running-python2 Python2 Audit | 19:48 |
| clarkb | fungi: jitsi meet supports that and I think we can even use the same number | 19:48 |
| clarkb | fungi: but that is new config we need to sort out | 19:48 |
| clarkb | (I don't know how it maps phone calls to meeting rooms as an example) | 19:49 |
| clarkb | One thing that was missing from the virtual event was unwind/decompression time | 19:49 |
| fungi | yeah, i figured it was something we could add | 19:49 |
| clarkb | at the in person events there are game nights and dinner with people | 19:50 |
| clarkb | I was wondering if anyone was interested in trying some virtual form of that | 19:50 |
| clarkb | more likely to be game night than dinner :) | 19:50 |
| fungi | also beer you don't have to pour yourself ;) | 19:50 |
| fungi | i guess i can get over pouring my own | 19:50 |
| clarkb | I've discovered hedgewars does remote multiplayer and maybe we can play a silly game of that with comms over meetpad | 19:50 |
| clarkb | its an open source clone of worms armageddon | 19:51 |
| clarkb | I'm open to other ideas or being told that there isn't sufficient interest | 19:51 |
| clarkb | Anything else to call out from the PTG? | 19:52 |
| clarkb | #topic Trusty Updates | 19:53 |
| *** openstack changes topic to "Trusty Updates (Meeting topic: infra)" | 19:53 | |
| clarkb | fungi: want to quickly recap the comodo cert situation? | 19:53 |
| fungi | sure | 19:53 |
| fungi | as of june 1, the old comodo/addtrust certificate authority ca cert expired | 19:54 |
| fungi | some of our sites used and still use certs which were validated through a chain including that as an intermediate | 19:54 |
| fungi | one in particular is openstackid.org | 19:55 |
| fungi | we discovered that on older python deployments, like that found on ubuntu trusty, the cert validation behavior of the requests module is to report a failure/exception if there is an expired cert in the chain bundle, even if another cert in the bundle is sufficient to validate the server's cert | 19:56 |
| fungi | this was causing people to be unable to log into refstack.openstack.org | 19:56 |
| fungi | it was ultimately "fixed" by updating the intermediate chain bundle on the openstackid.org server to no longer include the expired (and thus useless) addtrust cert | 19:57 |
| fungi | leaving only the newer sectigo cert | 19:58 |
| clarkb | and that is something we should apply to our other sectigo certs? | 19:58 |
| fungi | this matches the current chain bundle recommended by sectigo (the ca of record for our non-le certs obtained from namecheap) | 19:58 |
| fungi | it likely depends on what's out there accessing those sites | 19:59 |
| fungi | we can safely remove the old addtrust ca from all our intermediate bundles, but a lot of the copies i found are stale from before we started moving stuff to le | 19:59 |
| clarkb | ya so two layers of cleanup there I expect | 20:00 |
| fungi | so we could consider generally cleaning up old data in our hiera | 20:00 |
| clarkb | ++ | 20:00 |
| clarkb | and that takes us to the end of our alotted time | 20:00 |
| clarkb | thank you everyone | 20:00 |
| clarkb | Feel free to continue conversation in #opendev | 20:00 |
| fungi | if someone knows a programmatic way to identify those, that would be great | 20:00 |
| clarkb | but I'll end the meeting now to ensure people can eat breakfast/lunch or go to bed :) | 20:00 |
| clarkb | #endmeeting | 20:00 |
| fungi | thanks clarkb! | 20:00 |
| *** openstack changes topic to "Incident management and meetings for the OpenDev sysadmins; normal discussions are in #opendev" | 20:00 | |
| openstack | Meeting ended Tue Jun 9 20:00:53 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 20:00 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-06-09-19.01.html | 20:00 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-06-09-19.01.txt | 20:00 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-06-09-19.01.log.html | 20:00 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!