| clarkb | Anyone else here for the infra meeting? | 19:00 |
|---|---|---|
| diablo_rojo | o/ | 19:00 |
| ianw | o/ | 19:00 |
| clarkb | #startmeeting infra | 19:01 |
| opendevmeet | Meeting started Tue Jul 6 19:01:12 2021 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:01 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:01 |
| opendevmeet | The meeting name has been set to 'infra' | 19:01 |
| clarkb | #link http://lists.opendev.org/pipermail/service-discuss/2021-July/000264.html Our Agenda | 19:01 |
| clarkb | The agenda went out a bit late due to yesterday's holiday observance btu we do have an agenda :) | 19:01 |
| clarkb | #topic Announcements | 19:01 |
| clarkb | July 18 the Gerrit server will be upgraded. Update your firewall rules now if you need to do that (details are on the service-discuss mailing list) | 19:02 |
| clarkb | I've also got this info going out on the foundation newsletter this week to try and spread the word | 19:02 |
| clarkb | #topic Actions from last meeting | 19:02 |
| clarkb | #link http://eavesdrop.openstack.org/meetings/infra/2021/infra.2021-06-29-19.01.txt minutes from last meeting | 19:02 |
| clarkb | I had an action to talk to the openstack tc about the next steps for the ELK stack. I have done this and been asked to bring it up at the TC meeting on Thursday. I updated their agenda and plan to be there to discuss the subject | 19:03 |
| clarkb | It sounds like the board has asked them for timelines and a few details which I think we can help with too | 19:03 |
| clarkb | #action someone write spec to replace Cacti with Prometheus | 19:03 |
| clarkb | I don't think this has happend yet. But its possible I may have time for that this week looking at my current todo list. We'll see | 19:04 |
| fungi | someone is falling down on the job | 19:04 |
| fungi | it's times like this i'm glad to be noone | 19:04 |
| clarkb | careful we can make you a somebody | 19:04 |
| fungi | oof | 19:05 |
| clarkb | #topic Topics | 19:05 |
| clarkb | #topic Gerrit Account Cleanup | 19:05 |
| clarkb | This morning I retired 176 accounts that we identified as unused or unlikely to be unused | 19:05 |
| fungi | yes! | 19:06 |
| * fungi throws a quick party | 19:06 | |
| clarkb | This was based on account activity, age, and the situation the conflicting account is in. In many cases we could see the accounts haven't been used in almost a decade or one account was used then another took over | 19:06 |
| clarkb | That should leave us with about 80 accounts where the situation is more complicated and we'll try to reach out to users for those. | 19:06 |
| clarkb | The next step for these 176 is to wait 2 or 3 weeks then once we've given it time for people to raise any alarms we can remove the conflicting external ids from the retired accounts | 19:07 |
| fungi | as usual, if folks complain their account has broken, we should probably start by grepping the id from the logs you saved, yeah? | 19:07 |
| clarkb | I'd like to start reaching out to individuals in the ~80 remaining while we wait on that too | 19:07 |
| clarkb | fungi: yes exactly. Then you should be able to revert the commit I pushed to refs/users/xy/abxy to set them back to the way they were | 19:07 |
| fungi | awesome. thanks for confirming | 19:08 |
| clarkb | I'm 99% sure I tested this and it is only the external ids refs where gerrit will reject the changes. reverts to the refs/users/* refs are fine | 19:08 |
| clarkb | I did have a small network hiccup when retiring users so I had to rerun my retirement script for one user. That is the only oddity in the logs | 19:08 |
| clarkb | but the revert process doesn't change for that user | 19:09 |
| clarkb | #topic Review Upgrade | 19:10 |
| clarkb | This has been announced for the end of day July 18 UTC time | 19:10 |
| clarkb | as mentioned before I've got this info going out on the foundation newsletter to help advertise it. | 19:10 |
| clarkb | ianw: are there changes we should be reviewing now (like the SSHFP cleanups?) | 19:10 |
| fungi | this is the point in the meeting where i realize i'm actually away from home visiting extended family when the upgrade is scheduled to happen | 19:10 |
| fungi | but i'll try to be online anyway, their internet connectivity willing | 19:11 |
| ianw | yeah, if i could get eyes on | 19:11 |
| clarkb | fungi: I expect we'll be fine. ianw and I can be around | 19:11 |
| clarkb | but the help is appreciated | 19:11 |
| ianw | #link https://etherpad.opendev.org/p/gerrit-upgrade-2021 | 19:11 |
| ianw | and particularly the two pre-merge things | 19:11 |
| fungi | i've left a few notes on there, but will go over it more closely and check out the linked changes | 19:12 |
| ianw | ahh, thank you, i will go through comments again today | 19:12 |
| clarkb | ya I'ev gone over it too, but should go over it again and make sure I've reviewed the chagnes and followed up on any comments to the process | 19:12 |
| clarkb | One thought I had was that we should maybe land https://review.opendev.org/c/opendev/system-config/+/799225 to update our gerrit image and fix the lp blueprints integration then when we do the upgrade we'll know we have the same happy image | 19:13 |
| clarkb | er I mean land that then do a quick restart on prod before the upgrade (like this week?) | 19:13 |
| fungi | #link https://review.opendev.org/799124 Good riddance to track-upstream and its cronjob | 19:14 |
| fungi | that's related, since the new server seems to be opaquely failing to actually run its cronjob | 19:14 |
| clarkb | ianw: ^ if you don't think those two changes are a problem for pre merge activities maybe go ahead and add them to the etherpad? Or let me know and I can add them | 19:14 |
| ianw | ok, will look | 19:15 |
| ianw | i think maybe the switch of zuul to review01.opendev.org will require a zuul restart? | 19:15 |
| ianw | so it might be a good time to pull in the updated image too | 19:16 |
| clarkb | yes updates to that portion of the zuul config will require a zuul restart | 19:16 |
| ianw | i can do that on my monday when it's quiet | 19:16 |
| clarkb | ok | 19:16 |
| ianw | that gives it a week to bake in | 19:17 |
| clarkb | might need to coordinate with corvus on zuul restarts as there is a lot of chrun on the zuul side and we may have to do a full restart to safely update depending on the changes that land | 19:17 |
| clarkb | (zuulv5 development is full steam ahead) | 19:17 |
| ianw | indeed it is! | 19:17 |
| clarkb | Anything else on this topic? | 19:18 |
| ianw | not from me | 19:19 |
| clarkb | #topic Draft matrix spec | 19:19 |
| clarkb | A few of us had a call with Element Matrix Services (EMS) last week to discuss the possibility of using their hosted EMS platform | 19:19 |
| clarkb | From what I understood they didn't have any problems with us doing the slightly hacky setup to only maintain admins and bot users on our server then have users use matrix.org or their own homeservers | 19:20 |
| fungi | i have a feeling they're happy to see open source communities making use of it | 19:21 |
| clarkb | Their suggestion to us for next steps and getting started is to spin up a trial instance on the lowest tier of their service. Get things set up and start interacting with it. Then if we want to use the slightly more expensive silver version (they think this will be our best choice) we can upgrade to that painlessly | 19:21 |
| clarkb | fungi: yup they were super helpful in talking us through this | 19:21 |
| clarkb | I then talked to jbryce at the foundation about this and he didn't think this would be a problem. I need to coorindate with him to set up the accoutn and configure the payment details. Hopign to do that this week | 19:22 |
| fungi | what does silver get us over the entry level? | 19:22 |
| fungi | or is silver the lowest tier paid option? | 19:22 |
| clarkb | fungi: Nickel is the lowest option. Silver is second lowest | 19:22 |
| fungi | i see you said "trial" so i suppose that's time-limited? | 19:22 |
| clarkb | https://element.io/pricing | 19:22 |
| fungi | aww, missed opportunity there was no "dime" to go with "nickel" | 19:23 |
| clarkb | ya nickel is free for the first month as a trial thn you start paying for it | 19:23 |
| clarkb | my understanding over why silver would probably be best was simply a matter of scale | 19:23 |
| clarkb | we would probably be ok with nickel for zuul but as we grow that can change | 19:23 |
| clarkb | it is something to figure out as we go and interact with it | 19:24 |
| clarkb | They also noted that if we don't want our instance hosted in sweden you have to pay for Gold or higher but we don't think that is a problem | 19:24 |
| corvus | oh hai | 19:24 |
| clarkb | On our end the spec got updated to reflect the plan to try EMS | 19:24 |
| clarkb | #link https://review.opendev.org/796156 | 19:24 |
| fungi | i like sweden | 19:24 |
| clarkb | corvus: hello, I was just recounting what we learned from our call with EMS | 19:24 |
| clarkb | and noted that I talked to jbryce about it and he seemed happy with it. Now I need to coordinate with him to get an accoutn created with appropriate payment details | 19:25 |
| ianw | when it says "active user" that means people with a @user:opendev.org address? | 19:25 |
| corvus | lgtm | 19:25 |
| corvus | ianw: yes, so basically, our bots/admin accounts | 19:25 |
| clarkb | ianw: yes, and you can actually have a bunch of inactive versions of that in the system. They differentiate between actually active and you can log in and cold storage | 19:25 |
| clarkb | (not sure the active vs inactive designation will end up being useful for us but it means you can create accounts pretty safely and then just be careful about what you activate) | 19:26 |
| fungi | i guess if we needed occasional-use admin accounts that could come up | 19:26 |
| corvus | clarkb: i think i have 2 questions: 1) how to proceed with spec approval? 2) do you want to wait for that before setting up the account/server, or go ahead and get started on that and we can start working on bots, etc? | 19:27 |
| clarkb | corvus: I'd like to go ahead and get started with the account/server creation since we may learn something important doing that. | 19:27 |
| corvus | fungi: yeah, i sort of feel like a single admin account that's either used by a bot or one of us manually when required will probably be fine for things like setting up rooms, etc. | 19:27 |
| fungi | agreed, from what little i know so far | 19:28 |
| clarkb | corvus: but then once the server is up and spec updated to accomodate any new info and no major issues pop up I think we can land the spec? | 19:28 |
| corvus | clarkb: cool; everything about this can be reversed easily right up until we ask people in #zuul to move, so we have a lot of leeway if we're okay being casual about it. | 19:28 |
| clarkb | corvus: maybe give the trial a couple of weeks and then land the spec? | 19:28 |
| clarkb | corvus: good to know | 19:28 |
| clarkb | also mordred moved homeservers recently and that seems to have gone reasonably well | 19:28 |
| corvus | yeah, mordred also did a test of moving a room from his old homeserver to the new; even that worked without a hitch | 19:29 |
| mordred | I moved a channel I'd created ... yeah that ^^ | 19:29 |
| mordred | now - moving homeservers was a bit more involved and didn't really transition state | 19:29 |
| clarkb | I think I'm mostly worried about finding something about EMS that is a deal breaker for us and deciding we need to run it ourselves which will have a big impact on the spec | 19:29 |
| clarkb | but once we are past that step I think we should land the spec | 19:29 |
| clarkb | s/step/concern/ | 19:29 |
| fungi | i keep meaning to set up one since i have a private inspircd with some semi-used channels i'd like to add a bridge for eventually, and could have a vanity username that way too | 19:29 |
| mordred | from a user account pov - basically I had @mordred:waterwanders.com and I created @mordred:inaugust.com then invited @mordred:inaugust.com to anything that @mordred:waterwanders.com was in | 19:30 |
| mordred | but moving the room from waterwanders homserver to inaugust homeserver went amazing | 19:30 |
| clarkb | It does seem like the plan is congealing which is nice. I'll have to review the spec properly to indicate that | 19:30 |
| mordred | I believe if we have a homeserver with EMS and we decide at a point in the future we need to run it ourselves they can work with us to export the data and do an actual move | 19:31 |
| corvus | i was literally like "what's mordred talking about, this room is on inaugust" oh yeah, it didn't use to be. ;) | 19:31 |
| fungi | i guess the irc bridges work by emulating an irc server and networking with existing servers on the same irc network? | 19:31 |
| mordred | that seemed like a lot to ask them for just my little homeserver, so I didn't do it :) | 19:31 |
| corvus | fungi: i'm not 100% sure; but https://github.com/matrix-org/matrix-appservice-irc is apparently the software | 19:32 |
| clarkb | fungi: I suspect that the integartion is less coupled than that. They probably get connection limit exceptions for their bridge in the network then just emulate being a bunch of clients | 19:32 |
| fungi | ahh, thanks! i'll give that a thorough look | 19:32 |
| clarkb | Alright anything else to talk about on this subject? | 19:33 |
| corvus | oh 1 thing | 19:33 |
| corvus | feel free to sign up to make a replacement irc bot (see the spec) | 19:33 |
| corvus | eavesdrop/statusbot are available (tristanC wrote a gerritbot) | 19:34 |
| corvus | meetbot too, but is not needed for zuul | 19:34 |
| corvus | clarkb: otherwise, next steps seem clear to me | 19:34 |
| clarkb | and for eavesdrop we may not even need channel logging if we can just grab those directly from the matrix server | 19:34 |
| fungi | is there still any benefit to merging the bots into a single codebase? | 19:34 |
| clarkb | fungi: I'm not sure I know enough to say at this point :) | 19:35 |
| fungi | at a minimum we ought to at least shoot for significant code reuse | 19:35 |
| corvus | clarkb: probably the easiest way to do that though is to have a bot account join a room | 19:35 |
| clarkb | corvus: ah makes sense since that is the "api" | 19:35 |
| corvus | ultimately, i think having something writing an html file to disk is still useful for search engine indexing | 19:35 |
| corvus | (so even if an individual user can use the in-client search feature, if we value the indexing, we should have an eavesdrop bot) | 19:36 |
| corvus | (if we don't value search engine indexing, then, er, maybe we drop that :) | 19:36 |
| mordred | yeah. history is great - but having browsable html archives has been nice so far | 19:36 |
| ianw | it's always handy to be able to link to a prior conversation | 19:36 |
| mordred | yeah. I mean - you can do web links to matrix history - but they take you to a matrix client | 19:37 |
| clarkb | oh in that case ++ to having separate html | 19:37 |
| corvus | so yeah, that's a neat feature, but separately the plain html archive is still nice | 19:37 |
| mordred | and that's maybe a bit heavy for wanting to reference in some other context | 19:37 |
| mordred | ++ | 19:37 |
| clarkb | We have a few more topics to get to so lets move on | 19:38 |
| clarkb | #topic Gitea01 Backups | 19:38 |
| corvus | https://matrix.to/#/!eitSLAJcQKeehLruKf:matrix.org/$AB4j0_Z_HOn_pOAWfXdHSI-eHvDE_MMlJD2za1-kspU?via=matrix.org | 19:38 |
| fungi | seems the network connectivity between vexxhost regions is stil broken | 19:39 |
| clarkb | These continue to not be urgent to fix because we haven't done any recent project renames. However, ianw was looking at the issue any we plan to do renames in a few weeks | 19:39 |
| clarkb | fungi: ianw: want to fill us in on what you found? | 19:39 |
| fungi | short story is that some ipv6 addresses in sjc1 can't communicate with some ip addresses in ca-ymq-1 over some protocols | 19:39 |
| ianw | yeah, there was that | 19:40 |
| clarkb | ipv4 is fine though? | 19:40 |
| ianw | but the backup does seem to be daily running. i guess it's falling back to ipv4? | 19:40 |
| fungi | as if flows are being load balanced between routers at layer 4 and at least one router has a stray route with a greedy/short prefix | 19:40 |
| clarkb | ianw: well it is emailing us about the failures as recently as yesterday | 19:40 |
| ianw | the file-system backup. but then the db part fails | 19:40 |
| clarkb | oh got it | 19:40 |
| clarkb | I think well behaved applications are expected to fallback to ipv4 if v6 doesn't work | 19:41 |
| ianw | i found some dump options that looked promising, but i have to admit i got sidetracked on the ipv6 | 19:41 |
| clarkb | openssh is probably well behaved in this way | 19:41 |
| ianw | fungi: you double checked the ipv6 thing right? | 19:42 |
| fungi | yeah, if memory serves, the !h is being returned by the first hop in ca-ymq-1 | 19:42 |
| fungi | which leads me to suspect the core routing in that region, but it's all a black box to me. mnaser was looking into it | 19:43 |
| ianw | i'll ping again, i mentioned something at the end of last week | 19:43 |
| clarkb | sounds good. | 19:43 |
| clarkb | #topic Gerrit project renames | 19:44 |
| clarkb | #link https://review.opendev.org/797990 rename playbook updates | 19:44 |
| clarkb | That is a change we'll need to have in before we do renames. We're still a few weeks away from our week after the gerrit server upgrade so not in a rush but wanted to point it out | 19:44 |
| clarkb | If we can get that reviewed that would be great | 19:44 |
| clarkb | And if you know of any additional ernames that should be considered now is the time to get them on the list | 19:45 |
| fungi | all of osf/* should probably be renamed to openinfra/* but that's not urgent and i still need to talk to folks at the foundation about it | 19:45 |
| clarkb | fungi: ok | 19:45 |
| clarkb | I'll mention the rename schedulign to the TC on thrusday too | 19:45 |
| fungi | good call | 19:46 |
| clarkb | #topic Should we change our meeting time? | 19:46 |
| clarkb | It occurred to me that this meeting time is not very good for ianw (at least I don't think it is) and ianw is one of the primary participants. | 19:47 |
| clarkb | frickler mentions that they lurk the meetings to follow along even though they don't actively participate | 19:47 |
| clarkb | I wanted to put it out there that I'm ok with trying to find a time that works better for others if that would be helpful | 19:47 |
| fungi | i'm happy to do whatever time others would prefer | 19:47 |
| * fungi has no life | 19:47 | |
| fungi | though also i expect to miss the next meeting, possibly the text two | 19:48 |
| clarkb | frickler will miss the next three as well and mentioned that may be a good tiem to experiment with non EU timezone friendly meetings | 19:49 |
| ianw | i don't mind, in (my) summer the meeting moves 6-7am which is perfectly fine. 7-8am usually involves family, and at the tail end of 8-9 i have to do school run | 19:49 |
| ianw | so from my POV earlier is better | 19:50 |
| clarkb | ok I wanted to double check. Since we have had changes to who participates and could move things around if it helps | 19:50 |
| clarkb | Sounds like the current time slot is fine though and we can keep it as is | 19:51 |
| clarkb | That was all I had | 19:51 |
| clarkb | #topic Open Discussion | 19:51 |
| clarkb | Anything else before our hour is up? | 19:51 |
| ianw | if i could get a couple of eyes on | 19:52 |
| ianw | #link https://review.opendev.org/c/opendev/system-config/+/798400 | 19:52 |
| ianw | that adds paste to ansible | 19:52 |
| clarkb | ++ I'll add that to my afternoon todo list. | 19:52 |
| ianw | what i'm really interested in is the mariadb container behind that, same bits as for gerrit | 19:52 |
| ianw | would be nice to have a bit more experience with it | 19:53 |
| clarkb | Last call :) Otherwise I'll let everyone go find breakfast/lunch/dinner | 19:54 |
| clarkb | #endmeeting | 19:55 |
| opendevmeet | Meeting ended Tue Jul 6 19:55:12 2021 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 19:55 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/infra/2021/infra.2021-07-06-19.01.html | 19:55 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/infra/2021/infra.2021-07-06-19.01.txt | 19:55 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/infra/2021/infra.2021-07-06-19.01.log.html | 19:55 |
| clarkb | Thank you everyone! | 19:55 |
| clarkb | we'll see you here next week | 19:55 |
| clarkb | And as always feel free to reach out in #opendev or service-discuss@lists.opendev.org anytime to bring discussion up. We aren't beholden to this hour | 19:55 |
| fungi | thanks clarkb! | 19:57 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!