| *** hamalq has joined #opendev-meeting | 01:16 | |
| *** hamalq has quit IRC | 01:18 | |
| *** hamalq has joined #opendev-meeting | 01:33 | |
| *** hamalq has quit IRC | 01:39 | |
| *** hamalq has joined #opendev-meeting | 02:32 | |
| *** hamalq has quit IRC | 02:36 | |
| *** hamalq has joined #opendev-meeting | 02:48 | |
| *** hamalq has quit IRC | 02:52 | |
| *** hamalq has joined #opendev-meeting | 03:25 | |
| *** hamalq has quit IRC | 03:30 | |
| *** hamalq has joined #opendev-meeting | 03:54 | |
| *** hamalq has quit IRC | 03:58 | |
| *** hamalq has joined #opendev-meeting | 04:10 | |
| *** hamalq has quit IRC | 04:15 | |
| *** hamalq has joined #opendev-meeting | 06:46 | |
| *** hamalq has quit IRC | 06:47 | |
| *** hamalq has joined #opendev-meeting | 07:03 | |
| *** hamalq has quit IRC | 07:08 | |
| *** hamalq has joined #opendev-meeting | 08:08 | |
| *** hamalq has quit IRC | 08:12 | |
| *** hamalq has joined #opendev-meeting | 08:24 | |
| *** hamalq has quit IRC | 08:29 | |
| *** hamalq has joined #opendev-meeting | 08:30 | |
| *** hamalq has quit IRC | 08:30 | |
| *** hamalq has joined #opendev-meeting | 08:46 | |
| *** hamalq has quit IRC | 08:51 | |
| *** SotK has quit IRC | 09:08 | |
| *** tobiash has quit IRC | 09:08 | |
| *** SotK has joined #opendev-meeting | 09:08 | |
| *** tobiash has joined #opendev-meeting | 09:12 | |
| *** hamalq has joined #opendev-meeting | 09:21 | |
| *** hamalq has quit IRC | 09:26 | |
| *** hamalq has joined #opendev-meeting | 12:30 | |
| *** hamalq has quit IRC | 12:35 | |
| *** hamalq has joined #opendev-meeting | 12:46 | |
| *** hamalq has quit IRC | 12:51 | |
| *** hamalq has joined #opendev-meeting | 15:28 | |
| *** hamalq has quit IRC | 15:55 | |
| *** hamalq has joined #opendev-meeting | 15:55 | |
| *** hamalq_ has joined #opendev-meeting | 15:57 | |
| *** hamalq has quit IRC | 16:00 | |
| *** diablo_rojo has joined #opendev-meeting | 16:02 | |
| clarkb | meeting time. Anyone else here for our weekly gathering? | 18:59 |
|---|---|---|
| fungi | oh why not | 18:59 |
| fungi | ;) | 19:00 |
| mordred | nope. not here | 19:00 |
| mordred | you can't see me | 19:00 |
| fungi | indeed i cannot see you | 19:00 |
| clarkb | I can see what you type thoough | 19:00 |
| corvus | o/ | 19:00 |
| mordred | clarkb is inside of my keyboard!!! | 19:00 |
| frickler | o/ | 19:01 |
| fungi | mordred: you should clean your keyboard more often | 19:01 |
| clarkb | #startmeeting infra | 19:01 |
| openstack | Meeting started Tue Jun 23 19:01:10 2020 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:01 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:01 |
| *** openstack changes topic to " (Meeting topic: infra)" | 19:01 | |
| openstack | The meeting name has been set to 'infra' | 19:01 |
| clarkb | #link http://lists.opendev.org/pipermail/service-discuss/2020-June/000042.html Our Agenda | 19:01 |
| clarkb | #topic Announcements | 19:01 |
| *** openstack changes topic to "Announcements (Meeting topic: infra)" | 19:01 | |
| ianw | o/ | 19:02 |
| clarkb | On Thursday the OpenStack Foundation is doing two rounds of community updates. One for more europe friendly timezones and the other for asia pacific timezones. The Americas are sort of stuck in the middle | 19:02 |
| clarkb | feel free to join those if interested | 19:02 |
| clarkb | (though really the audience is people not involved in daily happenings so maybe boring for you all :) ) | 19:02 |
| clarkb | also that is relative to my Thursday. Local day of week may differ | 19:03 |
| clarkb | #topic Actions from last meeting | 19:04 |
| *** openstack changes topic to "Actions from last meeting (Meeting topic: infra)" | 19:04 | |
| clarkb | #link http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-06-16-19.01.txt minutes from last meeting | 19:04 |
| clarkb | We didn't record any actions | 19:04 |
| clarkb | #topic Specs approval | 19:04 |
| *** openstack changes topic to "Specs approval (Meeting topic: infra)" | 19:04 | |
| clarkb | #link https://review.opendev.org/#/c/731838/ Authentication broker service | 19:04 |
| clarkb | Still not ready for approval but worth calling out | 19:04 |
| fungi | i've started to pick it back up, haven't pushed a new patchset though | 19:04 |
| clarkb | fungi: thanks. Probably worth a call out when a new patchset does show up so we can take a look quickly | 19:05 |
| fungi | still digestnig recent comments, but more comments don't need to wait for me to update the spec either | 19:05 |
| clarkb | this is true | 19:05 |
| clarkb | #topic Priority Efforts | 19:06 |
| *** openstack changes topic to "Priority Efforts (Meeting topic: infra)" | 19:06 | |
| clarkb | #topic Update Config Management | 19:06 |
| *** openstack changes topic to "Update Config Management (Meeting topic: infra)" | 19:06 | |
| clarkb | haven't seen a whole lot on this front, but I've also been distracted with image updates and openstack testing stuff | 19:06 |
| clarkb | Is there anything worth calling out on this subject? Maybe the plan for reducing "run all the jobs when inventory updates" problems? | 19:07 |
| clarkb | mordred: ^ that caught corvus again on Friday when trying to land the zk config updates (which eventually applied and everything was fine but the number of jobs was unexpected) | 19:07 |
| fungi | ianw has started work on containerizing our grafana deployment | 19:08 |
| clarkb | ah cool. I ninja updated my local copy of the agenda to talk about grafana's nodepool dashboards in a bit too | 19:08 |
| ianw | can update in there, it is coming together i think | 19:09 |
| fungi | i also have a half-baked change underway to move our reprepro mirroring from puppet to ansible, i need to pick that back up and hack on it some more (unless anyone's just dying to take it over) | 19:09 |
| clarkb | mordred: is step0 there getting the split up puppet else changes landed? | 19:09 |
| clarkb | mordred: and if so are those ready for review? | 19:09 |
| mordred | not even | 19:10 |
| mordred | I think step0 can actually be just copying over the inventory/ file matchers from the system-config-run jobs | 19:11 |
| mordred | we have smaller file matchers for them already | 19:11 |
| clarkb | #info Copy file matchers from system-config-run jobs to infra-prod jobs to reduce number of jobs that run when inventory/ is updated | 19:12 |
| fungi | another good one to tackle soon might be storyboard... mordred's already added image building jobs for it, so assuming those are functional the actual deployment ansible for them might not be too hard (though there are rather a lot of different contaniers associated with it) | 19:12 |
| clarkb | fungi: that will also convert us to python3 for that service which would be nice | 19:13 |
| clarkb | Alright last call on config management changes | 19:13 |
| * diablo_rojo sneaks in the back and sits down | 19:13 | |
| corvus | wait we can sit during this meeting? | 19:14 |
| clarkb | I'm sitting | 19:14 |
| fungi | i thought it was a standing meeting | 19:14 |
| clarkb | sometimes I stand | 19:14 |
| fungi | but clarkb is a chair | 19:14 |
| fungi | let's table this | 19:14 |
| clarkb | #topic OpenDev | 19:14 |
| *** openstack changes topic to "OpenDev (Meeting topic: infra)" | 19:14 | |
| clarkb | First up we upgraded Gitea to version 1.12.0 | 19:14 |
| clarkb | This includes caching of info that Gitea uses to render repo UI pages which should speed those page loads up quite a bit. But its still a cache so the first load is still slow | 19:15 |
| clarkb | we've seen this help quite a bit for repos like nova already though | 19:15 |
| clarkb | I had some small concern that the caching would increasememory usage on those servers but I've not seen that being a problem yet according to cacti | 19:16 |
| clarkb | #link http://lists.opendev.org/pipermail/service-discuss/2020-May/000026.html Advisory Board thread. | 19:16 |
| clarkb | I've bumped this thread and am setting a soft deadline for July 2 where we'll proceed with the volunteers we have at that point | 19:16 |
| clarkb | soft deadline because we can always add new members later, but I didn't want to sit in this holding pattern forever | 19:17 |
| clarkb | The last OpenDev item I had was related to git branches. The topic of potentially renaming branches has come up in a couple of contexts and I wrote an etherpad with details on what that would mean for us as a hosting platform and our hosted repos | 19:18 |
| clarkb | https://etherpad.opendev.org/p/opendev-git-branches | 19:18 |
| clarkb | There are some potentially painful technical challenges which that tries to call out so that we can provide reasonable guidance to people that may consider such changes | 19:19 |
| clarkb | Anything else to talk about on the subject of OpenDev? or does anyone want to dig into anything above? | 19:20 |
| clarkb | #topic General topics | 19:21 |
| *** openstack changes topic to "General topics (Meeting topic: infra)" | 19:21 | |
| clarkb | #topic Grafana Nodepool Graphs | 19:21 |
| *** openstack changes topic to "Grafana Nodepool Graphs (Meeting topic: infra)" | 19:21 | |
| clarkb | We've discovered that we're no longer producing templated nodepool provider graphs taht function | 19:22 |
| clarkb | looking at the dashboard json it appears that we aren't setting a data source properly. I wrote a change to the nodepool-ovh dashboard to force the data source and that doesn't seem ot have helped | 19:22 |
| clarkb | The change I wrote did end up switching the data source value from null to OpenStack though | 19:23 |
| clarkb | which maybe means the problem is unrelated to data sources | 19:23 |
| clarkb | ianw: ^ maybe it makes more sense to debug once we've got the deployment tooling switched out? | 19:23 |
| fungi | ossh grafana | 19:24 |
| ianw | yeah, so i saw that and started to debug too and i guess yak shaving took over | 19:24 |
| fungi | heh, this is not my shell terminal ;) | 19:24 |
| ianw | however, i think it probably makes more sense to start debugging latest grafana via containers | 19:24 |
| ianw | i have it mostly working, building a container based on upstream grafana and including grafyaml in it | 19:25 |
| ianw | it's easy to deploy locally and should work the same in production | 19:26 |
| clarkb | that is likely to aid in debugging | 19:26 |
| clarkb | graphite is publicly accessible so we can point at production data easily | 19:26 |
| fungi | is the suspicion at this point that our regression is due to the recent grafana 7.x releases? | 19:26 |
| ianw | i think so | 19:26 |
| clarkb | fungi: ya I'm assuming that grafyaml isn't supplying the necessary info to make the templating work | 19:27 |
| ianw | #link https://review.opendev.org/737397 | 19:27 |
| ianw | that's the base container | 19:27 |
| clarkb | what our grafyaml does is tell grafana to query graphite for a list of nodepool provider regions. Then using that list it produces graphs for each entry in the list | 19:27 |
| ianw | #link https://review.opendev.org/737406 | 19:27 |
| clarkb | from what I can tell we aren't producing a valid input list to the graphs | 19:27 |
| ianw | that will test it. both are really active wip | 19:27 |
| clarkb | ok, I'll try not to worry about it too much until we've got the deployment stuff redone as that will simplify debugging | 19:28 |
| ianw | i agree, let's work from that common base | 19:28 |
| clarkb | #topic Etherpad Upgrade to 1.8.4 or 1.8.5 | 19:30 |
| *** openstack changes topic to "Etherpad Upgrade to 1.8.4 or 1.8.5 (Meeting topic: infra)" | 19:30 | |
| clarkb | Bringing this up to say I'm planning on holding off until the end of next week since the opendev event next week will use the etherpad | 19:30 |
| clarkb | once that is done it should be very safe to use a hacky 1.8.4 :) | 19:30 |
| clarkb | the change to do the upgrade is WIP'd with a similar message and shouldn't land early | 19:31 |
| fungi | 1.8.4 is still the current release at this time | 19:31 |
| clarkb | yes | 19:31 |
| fungi | i suppose "1.8.5" here could be a stand-in for tip of develop branch | 19:31 |
| clarkb | ya 1.next might be most accurate | 19:32 |
| clarkb | if something after 1.8.4 arrives before end of next week I'll respin to test and deploy that. Otherwise I'll land our change as is on 1.8.4 with local css fix | 19:32 |
| mordred | ++ | 19:33 |
| clarkb | #topic DNS cleanup | 19:34 |
| *** openstack changes topic to "DNS cleanup (Meeting topic: infra)" | 19:34 | |
| clarkb | #link https://etherpad.opendev.org/p/rax-dns-openstack-org First pass of record removals has been done. Could use a second pass. | 19:34 |
| clarkb | ianw thank you for putting this together. From what I can see its all working as expected. I did a first pass of removals based on what was noted on the etherpad too. | 19:34 |
| clarkb | Since then fungi has annotated more things on the etherpad and I think we need a second pass at cleanup. I was planning to do that todayish | 19:34 |
| clarkb | Calling this out so that if anyone else wants to do a pass they do it soon and catch it in my next set of removals | 19:36 |
| clarkb | I should also share it with the foundation sysadmins again and see if they want to remove anything | 19:36 |
| clarkb | #topic Getting more stuff off of python2 | 19:37 |
| *** openstack changes topic to "Getting more stuff off of python2 (Meeting topic: infra)" | 19:37 | |
| clarkb | #link https://etherpad.opendev.org/p/opendev-tools-still-running-python2 | 19:37 |
| clarkb | I ran out of steam on this last week, but wanted to call it out again in case others have notes to add about things they know need python2 attention | 19:37 |
| clarkb | Not incredibly urgent, but crowd sourcing tends to help with this sort of problem space | 19:38 |
| clarkb | #topic Wiki Upgrade | 19:38 |
| *** openstack changes topic to "Wiki Upgrade (Meeting topic: infra)" | 19:38 | |
| clarkb | fungi anything new on the wiki upgrade? | 19:38 |
| fungi | nope! | 19:38 |
| clarkb | #topic Open Discussion | 19:39 |
| *** openstack changes topic to "Open Discussion (Meeting topic: infra)" | 19:39 | |
| mordred | please to review https://review.opendev.org/#/c/733967/ and https://review.opendev.org/#/c/737023/ | 19:39 |
| clarkb | As a general heads up I'm going to be helping with the opendev event next week and will be distracted during those hours. I'm also looking at taking July 6-10 as a staycation and will need someone else to chair the meeting on the 7th if we have a meeting | 19:40 |
| fungi | in case folks hadn't seen, a while back i audited our listserv mta logs and determined that the long-running qq.com spam flood to the -owner aliases has *finally* abated. the starlingx list owners have requested we start allowing messages to the -owner alias for their list again so i proposed https://review.opendev.org/729649 but more generally we could think about lifting all of the blackhole | 19:41 |
| fungi | aliases i think | 19:41 |
| fungi | they do still receive some random spam on a daily or weekly basis (roughly proportional to their list activity volume it seems) so just turning it back on for all lists without warning could catch some folks by surprise | 19:44 |
| fungi | we've also had issues in the past with people setting list owner addresses to e-mail service providers who happily report those moderation request messages as spam to blacklisting services, which creates unfortunate deliverability issues for us | 19:45 |
| clarkb | neat | 19:45 |
| fungi | anyway, just crossed my mind again, not sure if anybody has suggestions for how to go about that | 19:46 |
| clarkb | fungi: maybe we should reach out to list moderators and ask them if they'd like it to be toggled? | 19:46 |
| clarkb | we know who those people are and can ask directly I think | 19:46 |
| fungi | yeah, i thought about that. there are a lot, but the deduplicated list of them might not be so many | 19:47 |
| clarkb | Anything else? I'll give it another minute or two then call it | 19:47 |
| clarkb | thank you everyone for your time | 19:48 |
| fungi | oh, also it looks like a stale apache worker on static.opendev.org caused sslcheck to fire a warning about the cert for zuul-ci.org expiring in a month | 19:48 |
| fungi | can move discussion of what to do for that and when to #opendev though | 19:48 |
| clarkb | ++ | 19:49 |
| clarkb | #endmeeting | 19:49 |
| *** openstack changes topic to "Incident management and meetings for the OpenDev sysadmins; normal discussions are in #opendev" | 19:49 | |
| openstack | Meeting ended Tue Jun 23 19:49:14 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 19:49 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-06-23-19.01.html | 19:49 |
| clarkb | thanks again! | 19:49 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-06-23-19.01.txt | 19:49 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-06-23-19.01.log.html | 19:49 |
| diablo_rojo | Thanks clarkb! | 19:53 |
| fungi | thanks clarkb! | 19:54 |
| *** clarkb has quit IRC | 22:44 | |
| *** clarkb has joined #opendev-meeting | 22:58 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!