| *** hamalq has quit IRC | 01:15 | |
| *** hashar has joined #opendev-meeting | 07:37 | |
| *** sboyron has joined #opendev-meeting | 08:33 | |
| *** sboyron has quit IRC | 17:46 | |
| *** sboyron has joined #opendev-meeting | 17:48 | |
| *** hamalq has joined #opendev-meeting | 17:58 | |
| *** ianw_pto is now known as ianw | 18:58 | |
| clarkb | Alright is anyone else going to be around for the meeting? I realized I failed to send an agenda this morning and was busy with other meetings so one didn't go out | 19:00 |
|---|---|---|
| ianw | o/ | 19:00 |
| clarkb | I'll start a meeting to go over some things that are worth mentioning but then probably just open it up from there | 19:00 |
| clarkb | #startmeeting Infra | 19:01 |
| openstack | Meeting started Tue Jan 12 19:01:07 2021 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:01 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:01 |
| *** openstack changes topic to " (Meeting topic: Infra)" | 19:01 | |
| openstack | The meeting name has been set to 'infra' | 19:01 |
| clarkb | #topic Announcements | 19:01 |
| *** openstack changes topic to "Announcements (Meeting topic: Infra)" | 19:01 | |
| clarkb | The foundation's board of director elections are happening this week. If you are a foundation member you shoulve have received an email with your voting instructions in it | 19:01 |
| clarkb | Please take a few minutes to go and vote if you are able to | 19:02 |
| clarkb | #topic Actions from last meeting | 19:02 |
| *** openstack changes topic to "Actions from last meeting (Meeting topic: Infra)" | 19:02 | |
| clarkb | #link http://eavesdrop.openstack.org/meetings/infra/2021/infra.2021-01-05-19.01.txt minutes from last meeting | 19:02 |
| clarkb | corvus had an action to implement Gerrit WIP support in Zuul. I believe the change was written and I reviewed it. corvus if you are around has that merged yet? | 19:02 |
| fungi | if not, i'd appreciate a link so i can review | 19:03 |
| clarkb | ya trying to find that now | 19:03 |
| clarkb | #link https://review.opendev.org/c/zuul/zuul/+/769436 looks like it merged according to gerritbot which is where I Found the link | 19:03 |
| fungi | awesome | 19:03 |
| fungi | so we need a restart to pick that up | 19:03 |
| clarkb | that means the next step for us in supporting WIP is to restart the zuul scheduler and then test again | 19:03 |
| clarkb | yup | 19:03 |
| fungi | at least the scheduler, presumably | 19:04 |
| fungi | ahh, you said those things | 19:04 |
| * fungi is a bit scattered today | 19:04 | |
| clarkb | me too | 19:04 |
| clarkb | #topic Priority Efforts | 19:04 |
| *** openstack changes topic to "Priority Efforts (Meeting topic: Infra)" | 19:04 | |
| clarkb | #topic OpenDev | 19:04 |
| *** openstack changes topic to "OpenDev (Meeting topic: Infra)" | 19:04 | |
| clarkb | First up it was pointed out that we were overdue for service coordinator nominations (and an election if necesasry) | 19:05 |
| clarkb | #link http://lists.opendev.org/pipermail/service-discuss/2021-January/000161.html | 19:05 |
| clarkb | I sent email dscribing what I think is a reasonable plan for addressing this miss | 19:05 |
| clarkb | if you think that plan is flawed in some way please respond to the mailing list and help us set up a new less flawed plan :) | 19:05 |
| clarkb | any other election feedback is also appreciated there. | 19:06 |
| clarkb | If you are interested in taking on the service coordinator role I'm happy to talk about it if you need more info before committing. Otherwise please send email to the service-discuss list nominating yourself | 19:06 |
| clarkb | as mentioned in the email I linked I've done it for a number of cycles now and think that new perspectives would be a good thing to have. | 19:07 |
| clarkb | I'm not saying I won't run again if necessary, but do strongly feel that having a bit more rotation would be a good thing | 19:07 |
| fungi | prepare for zbr to volunteer | 19:07 |
| fungi | zbr: you know you want to! | 19:08 |
| clarkb | the email I sent also sets up a week for electiosn if they become necessary. And to avoid this problem of missing them in the future I set out dates for the next set of nominations and elections | 19:08 |
| clarkb | I'll add them into my calendar reminders if later if no one ends up objecting to that proposed plan | 19:09 |
| clarkb | The opendev project update for the foundation's annual report is basically finalized now | 19:09 |
| clarkb | #link https://etherpad.opendev.org/p/opendev-2020-annual-report | 19:09 |
| clarkb | I think those are due tomorrow so if there are important edits get them in now (or let me know and I can make them) | 19:09 |
| clarkb | The last opendev topic I wanted to bring up was the gitea 1.13.1 upgrade. | 19:10 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/769226 | 19:10 |
| mordred | lgtm | 19:10 |
| clarkb | I think this is ready for serious consideration. There is a held gitea test node (somewhere I need to find it again via nodepool) running 1.13.1 to help confirm we're happy with it | 19:11 |
| clarkb | my biggest concern is that gitea 1.13 added some big new features like kanban boards to projects so want to make sure we're presenting a gitea that is consistent with our current setup | 19:11 |
| mordred | patch itself looks good - assuming the test node is solid | 19:12 |
| clarkb | ya I looked it over (and I think fungi did too?) and it seemed fine | 19:12 |
| fungi | i did, think i already +2'd | 19:12 |
| clarkb | if it does look good I should have plenty of time to watch it land and monitor it thursday | 19:12 |
| fungi | trying to get through some project-config backlog today, i noticed this one has a potential for global disruption but would like to merge and watch it closely after the meeting: | 19:13 |
| fungi | #link https://review.opendev.org/760495 Use internal mirror for RAX IAD/DFW | 19:13 |
| fungi | that switches the interface for mirror connections in two regions to use the second nic where bandwidth may be less constrained | 19:13 |
| clarkb | ++ | 19:13 |
| fungi | (we're already doing it that way in one) | 19:13 |
| mordred | ++ | 19:14 |
| mordred | might be worth simplifying that to "if nodepool.cloud == 'rax'" at this point | 19:14 |
| fungi | true | 19:14 |
| fungi | ianw: ^ wdyt? | 19:15 |
| mordred | not that I expect us to grow new rax regions or anything | 19:15 |
| fungi | i'm happy to tweak it before approving if folks prefer | 19:15 |
| ianw | yeah, i don't mind; we have used this before for switching in other clouds so it might be useful to keep the cloud/region append just as an example of what to do | 19:15 |
| fungi | i'm good with it as-is too | 19:16 |
| clarkb | why don't we land it as is then just to avoid unnecessary churn | 19:16 |
| fungi | wfm, will do | 19:17 |
| clarkb | #topic General Topics | 19:17 |
| *** openstack changes topic to "General Topics (Meeting topic: Infra)" | 19:17 | |
| clarkb | #topic Bup and Borg Backups | 19:17 |
| *** openstack changes topic to "Bup and Borg Backups (Meeting topic: Infra)" | 19:17 | |
| clarkb | ianw: this was on my list to check up on after the holidays. Are we completely off of bup at this point for new backups? | 19:18 |
| clarkb | and if so should I drop this item from our meeting agendas? | 19:18 |
| ianw | yep, since https://review.opendev.org/c/opendev/system-config/+/766300 | 19:18 |
| ianw | i still have to finish the cleanup with https://review.opendev.org/c/opendev/system-config/+/766630/ | 19:18 |
| ianw | i will work on that | 19:19 |
| clarkb | thanks | 19:19 |
| clarkb | and thank you for working on that | 19:19 |
| clarkb | we should be able to start looking at focal nodes now I think | 19:19 |
| clarkb | (since this was a big hold up for that iirc) | 19:19 |
| ianw | maybe keep it for one more week as i cleanup the old servers | 19:20 |
| clarkb | can do | 19:20 |
| fungi | i can't remember, was borg manually added to the wiki server? if not, i'll try to prioritize that | 19:20 |
| clarkb | fungi: I'm not sure | 19:20 |
| clarkb | probably not? | 19:20 |
| ianw | fungi: not sure either. i don't remember doing it. i can look into that | 19:20 |
| fungi | it was being backed up with bup | 19:21 |
| fungi | (still is afaik) | 19:21 |
| clarkb | it likely still is ya | 19:21 |
| clarkb | since the bup bits are still there iirc | 19:21 |
| clarkb | (we have to keep them around for backup retention anyway) | 19:21 |
| ianw | maybe give me an action item to confirm wiki being backed up so we don't forget | 19:22 |
| clarkb | #action ianw confirm wiki is still backed up after bup to borg migration | 19:22 |
| clarkb | #topic openstackid.org scale down | 19:23 |
| *** openstack changes topic to "openstackid.org scale down (Meeting topic: Infra)" | 19:23 | |
| clarkb | fungi and I conferred with smarcet to confirm that the desired running state for openstackid.org is to scale it down after we scaled it up for the summit | 19:24 |
| clarkb | fungi returned openstackid.org to its ansible + puppet managed state and I have shut down the two new servers in vexxhost and removed their A and AAAA dns records | 19:24 |
| clarkb | in a day or two when we're happy that we've cleaned things up without disruption I will delete them | 19:24 |
| clarkb | we also discovered that smarcet uses docker for openstackid development so it sounds like we can collaborate to convert that over to a ansible + docker-compose + docker deployment | 19:25 |
| clarkb | however, that is more of a "yes we can do that" idea at this point | 19:25 |
| clarkb | mostly an fyi on that since that service had some changes and now we've undone them. And now everyone else is caught up | 19:26 |
| clarkb | #topic Open Discussion | 19:26 |
| *** openstack changes topic to "Open Discussion (Meeting topic: Infra)" | 19:26 | |
| clarkb | That was what I had written down in notes really quickly after my previous meetings ended. We've got plenty of time to tlk about other topics if we need to | 19:26 |
| clarkb | oh! | 19:27 |
| clarkb | https://bugs.chromium.org/p/gerrit/issues/detail?id=13930 is worth pointing out | 19:27 |
| clarkb | I think I managed to figure out why fedora 33 users are still having trouble with new gerrit and rsa | 19:27 |
| clarkb | long story short is upstream openssh has only deprecated ssh-rsa for hostkey signature exchanges. Fedora has disabled it for hostkey stuff and for public key auth | 19:27 |
| clarkb | Gerrit does rsa-sha2-* just fine for hostkey stuff but it does not work for pubkey auth because that requires supporting the server-sig-algs kex extension in the server and gerrit's server doesn't seem to support that | 19:28 |
| corvus | oh sorry i got sucked into an issue | 19:28 |
| clarkb | fedora 33 users can work around this by enabling ssh-rsa or switching to an ed25119 or ecdsa key for auth | 19:29 |
| ianw | i feel like i'm using fedora 33 and it is working | 19:29 |
| ianw | ohhh, i'm using and ed25119 key | 19:29 |
| clarkb | ya this is specific to using rsa keys to auth | 19:29 |
| fungi | ianw: see, you anticipated this | 19:29 |
| clarkb | if fedora users have qusetions we can point them to that bug. I'm hoping upstream will say "oh thats an easy fix" and it will magically happen but I think it may be more involved | 19:30 |
| *** hashar has quit IRC | 19:30 | |
| ianw | heh, yep, istr having to merge some changes to our puppet to handle ed keys years ago :) | 19:30 |
| clarkb | in particular I think the proper way to fix this is to update mina upstream since other mina sshds will want the same fix | 19:30 |
| clarkb | eventually ssh clients should switch their fallback rsa pubkey auth type to rsa-sha2-something. But until that happens I expect this will be a problem for people | 19:31 |
| clarkb | related: if anyone knows fedora devs ^ it might be worth suggesting they make that switch | 19:31 |
| clarkb | since they are disabling the alternative | 19:31 |
| fungi | well, and also fedora could improve the situation by not still looking for sha-1 with ssh-rsa and instead trying sha-2 first | 19:31 |
| fungi | er, what you also just typed | 19:32 |
| clarkb | ya it feels like fedora's disabling of ssh-rsa missed an important step | 19:32 |
| clarkb | which was to not fallback to ssh-rsa when doing rsa pubkey auth talking to a server that doesn't do server-sig-algs | 19:32 |
| clarkb | https://tools.ietf.org/html/rfc8332#section-3.3 notes that this is the expected end state once rsa-sha2 is sufficiently ubiquitous (and it seems fedora is saying that it is) | 19:33 |
| ianw | is it https://bugzilla.redhat.com/show_bug.cgi?id=1881301 ? | 19:34 |
| openstack | bugzilla.redhat.com bug 1881301 in openssh "openssh-clients do not accept PubkeyAcceptedKeyTypes rsa-sha2-512/256" [Unspecified,Closed: errata] - Assigned to jjelen | 19:34 |
| clarkb | ianw: ya I think that is the bug on the fedora side | 19:35 |
| ianw | i'd like to get the zuul summary plugin going | 19:36 |
| ianw | the review stack is @ https://review.opendev.org/q/topic:%22gerrit-admin-user%22 | 19:36 |
| clarkb | ianw: and that plugin is hosted upstream now too right? | 19:37 |
| ianw | yes, that's right | 19:37 |
| clarkb | excellent I'll add that very high on the review todo list once I've got time to do that (probably tomorrow?) | 19:37 |
| ianw | ok, yeah the stuff underneath is to cleanup review-dev, then initalize and populate gerrit during testing, then add selenium testing and take screenshots, and then finally add the plugin | 19:38 |
| ianw | with a little bazelisk stuff for good measure :) | 19:39 |
| clarkb | alright anything else? | 19:41 |
| zbr | i am back | 19:41 |
| zbr | (reading backlog) | 19:41 |
| fungi | zbr: just me encouraging folks to make our service coordinator election an election this time | 19:42 |
| clarkb | I'll give zbr a coupel of minutes to catch up but then if that is it we can call it a meeting | 19:42 |
| fungi | (was the nick highlight i mean) | 19:42 |
| zbr | sure. ok to call the meeting off. | 19:45 |
| clarkb | thanks everyone. Sorry I missed the agenda. I'll do my best to not dismiss the alert until actually done in the future :) | 19:45 |
| clarkb | I think what happened was I habitually swiped it away when my phone made noise and had a thing pop up | 19:45 |
| clarkb | #endmeeting | 19:45 |
| *** openstack changes topic to "Incident management and meetings for the OpenDev sysadmins; normal discussions are in #opendev" | 19:45 | |
| openstack | Meeting ended Tue Jan 12 19:45:52 2021 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 19:45 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/infra/2021/infra.2021-01-12-19.01.html | 19:45 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/infra/2021/infra.2021-01-12-19.01.txt | 19:45 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/infra/2021/infra.2021-01-12-19.01.log.html | 19:45 |
| fungi | if my phone tried to tell me what to do, i'd probably chuck it out a window | 19:46 |
| zbr | as a side node, the RSA issue made me switch my main key to ssh-ed25519 | 19:47 |
| zbr | also podman remoting support no longer works with RSA. that was the second reason. | 19:47 |
| fungi | interesting | 19:48 |
| zbr | fedora can be reconfigured to work with rsa, but podman had not bugfix yet. | 19:48 |
| zbr | but few people are using the remoting support in it. | 19:48 |
| clarkb | aiui there really isn't any reason to stop using rsa except that it is old and has some gotchas around ti like sha1 that can be avoided (also wanting more key bits) | 19:48 |
| zbr | clarkb: i did not had any plan to replace it, i was more of forced to swap primary key and keep the rsa as fallback. | 19:49 |
| fungi | people are also scared that shor's algorithm will make quantum computers abel to factor large rsa keys | 19:49 |
| clarkb | ya fedora has definitely made it hard to use rsa now | 19:49 |
| zbr | mine was 4096, it was standing-off between others :D | 19:49 |
| ianw | i feel like if i'm understanding 1881301 fedora 33 has a patch for this (https://bugzilla.redhat.com/show_bug.cgi?id=1881301#c29) *but* you have to have set "update-crypto-policy --set=LEGACY"? | 19:50 |
| openstack | bugzilla.redhat.com bug 1881301 in openssh "openssh-clients do not accept PubkeyAcceptedKeyTypes rsa-sha2-512/256" [Unspecified,Closed: errata] - Assigned to jjelen | 19:50 |
| zbr | extra bonus: the new ssh-ed25519 pubkey is 10x shorter. | 19:50 |
| clarkb | ianw: right its reducing the security stance rather than changing the auth fallback to rsa-sha2-512 whihc would keep the same level of security they are trying toenforce | 19:51 |
| fungi | the flip side of that coin is "new" cryptography tends to have more potential flaws lurking somewhere inside while older algorithms have been vetted for longer by many more people | 19:52 |
| clarkb | ianw: if you look at that rfc link I posted it basically says "we will change this fallback default in clients once rsa-sha2 is ubiquitous enough" and my take on that is fedora is saying it is ubiquitous enough for them so they should change that fallback | 19:52 |
| zbr | yep, re-enabling it is a considerable PITA combines with a recurity risk | 19:52 |
| fungi | succinctly put, cryptography is often about "choosing between the devil you know and the devil you don't" | 19:53 |
| fungi | i' | 19:54 |
| fungi | i'm still reasonably comfortable relying on md5 in situations where the baseline is trusted. for comparing two untrusted inputs though it's now effectively useless | 19:55 |
| zbr | clarkb: you said something recently regarding some kind of possible sprint or at least some kind of joined effort on taking care of some tasks. just ping me when you know them. | 19:56 |
| fungi | but users don't want to have to think about the nuances of the problems they're addressing with cryptography, so newer primitives with fewer caveats (even if they may not be as well understood as their older counterparts) are generally winning out | 19:56 |
| fungi | zbr: great point, let's discuss in #opendev in case people aren't watching in here between meetings | 19:57 |
| zbr | clearly the rsa it was not well documented/advertised, i do not see mayself and average linux user and it took me a lot of time to decide between the other two other algorithms to use. | 19:58 |
| *** hashar has joined #opendev-meeting | 21:31 | |
| *** hashar has quit IRC | 23:15 | |
| *** sboyron has quit IRC | 23:45 | |
| *** sboyron has joined #opendev-meeting | 23:46 | |
| *** sboyron has quit IRC | 23:54 | |
| *** sboyron has joined #opendev-meeting | 23:56 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!