| *** hamalq has quit IRC | 02:25 | |
| *** hamalq has joined #opendev-meeting | 02:33 | |
| *** hamalq has quit IRC | 03:04 | |
| *** hamalq has joined #opendev-meeting | 15:10 | |
| *** hamalq_ has joined #opendev-meeting | 15:12 | |
| *** hamalq has quit IRC | 15:15 | |
| *** diablo_rojo has joined #opendev-meeting | 19:00 | |
| zbr | o/ | 19:00 |
|---|---|---|
| diablo_rojo | o/ | 19:00 |
| ianw | #startmeeting infra | 19:00 |
| openstack | Meeting started Tue Jul 7 19:00:33 2020 UTC and is due to finish in 60 minutes. The chair is ianw. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:00 |
| *** openstack changes topic to " (Meeting topic: infra)" | 19:00 | |
| openstack | The meeting name has been set to 'infra' | 19:00 |
| fungi | ahoy, mateys | 19:00 |
| zbr | ahoy | 19:01 |
| ianw | #topic Announcements | 19:01 |
| *** openstack changes topic to "Announcements (Meeting topic: infra)" | 19:01 | |
| corvus | aloha | 19:01 |
| ianw | no clarkb today, so i am copy/pasting the agenda #topics for a change | 19:01 |
| ianw | no other announcements of note | 19:02 |
| ianw | #topic Actions from last meeting | 19:02 |
| *** openstack changes topic to "Actions from last meeting (Meeting topic: infra)" | 19:02 | |
| ianw | #link http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-06-30-19.01.html minutes from last meeting | 19:02 |
| ianw | there were no action items, so we can keep moving on that | 19:02 |
| ianw | #topic Specs approval | 19:02 |
| *** openstack changes topic to "Specs approval (Meeting topic: infra)" | 19:02 | |
| ianw | #link https://review.opendev.org/#/c/731838/ Authentication broker service | 19:02 |
| ianw | this doesn't appear to have changed since last week, but seems to have a fair bit of comment | 19:03 |
| ianw | fungi: any further thoughts? | 19:03 |
| fungi | it was revised last week-ish | 19:03 |
| fungi | ready for more comments | 19:03 |
| fungi | i think all the comments thus far have been addressed | 19:04 |
| mordred | o/ | 19:04 |
| fungi | last week was a bit of a black hole for me what with the opendev large scale deployments conference, so it was probably actually week before last i revised it | 19:05 |
| fungi | anywah, bring on the comments | 19:05 |
| ianw | cool, so anyone who has an interest in, or knows anything about authentication should attach themselves to that | 19:05 |
| ianw | #topic Priority Efforts | 19:05 |
| *** openstack changes topic to "Priority Efforts (Meeting topic: infra)" | 19:06 | |
| ianw | #topic Opendev | 19:06 |
| *** openstack changes topic to "Opendev (Meeting topic: infra)" | 19:06 | |
| ianw | firstly looping back on the gitea pagination issues, it seems clarkb's patch was accepted | 19:06 |
| fungi | so maybe we can re-simplify manage-projects at next upgrade? | 19:07 |
| ianw | yeah, that was my question, are we tracking that somehow | 19:07 |
| ianw | #link https://github.com/go-gitea/gitea/pull/12057 gitea pagination fixes | 19:07 |
| mordred | I've been tracking it via the "clarkb mentions its progress" method | 19:08 |
| ianw | ok, i'll put next week's date on the topic and we can loop back on that | 19:08 |
| mordred | since they accepted it upstream, we could cherry-pick it into our image builds and re-simplify - or just wait | 19:08 |
| ianw | at next meeting | 19:08 |
| fungi | sounds good, it may be time to act on by then given their release cadence | 19:09 |
| corvus | i don't think the simplification is urgent | 19:09 |
| corvus | (iirc) | 19:09 |
| ianw | the ddos i have down as a separate topic, so we can discuss that there | 19:09 |
| fungi | i seem to have picked the wrong afternoon to attend a meeting from my patio, there are children having a birthday party at the rental house next door. there's now a bouncy castle, and i'm coveting it (maybe after a heavy round of disinfecting) | 19:10 |
| ianw | so the only other bit was governance changes here, of which i don't think there's new news since last week | 19:10 |
| fungi | i think it was any time after the first of this month we could start publishing a list of representatives for the advisory board, we just haven't gotten to it yet | 19:11 |
| fungi | people are still welcome to volunteer at any time anyway | 19:11 |
| ianw | ++ | 19:11 |
| ianw | #topic Update Config Management | 19:12 |
| *** openstack changes topic to "Update Config Management (Meeting topic: infra)" | 19:12 | |
| ianw | couple of things to get updates on | 19:12 |
| ianw | zuul-executors as containers : that's complete? i noticed they're still in emergency yesterday | 19:12 |
| mordred | last I knew there was another missing package in the container for afs - but I think that got fixed, yes? | 19:13 |
| ianw | i remember seeing something go by about the client package being a suggests | 19:14 |
| corvus | there's one more thing | 19:14 |
| corvus | we need gear | 19:14 |
| corvus | i think (haven't double checked) that ze01 is still disabled due to that | 19:14 |
| corvus | this one is trickier | 19:15 |
| mordred | yeah - ze01 is disabled | 19:15 |
| corvus | i feel less enthusastic about asking the zuul community to add gear to the zuul-exec images since nothing in zuul-jobs uses gear | 19:15 |
| corvus | (i recognize this is weird, since zuul itself uses gear) | 19:15 |
| corvus | the idea that has bubbled to the top of my list is actually to just vendor gear into our opendev base jobs | 19:15 |
| corvus | (this is for the logstash submit thing) | 19:16 |
| fungi | that seems like a fine compromise to me, at least | 19:16 |
| corvus | i also recognize that's icky. but it lets us avoid the weird conversation about including opendev-specific stuff in zuul images, and avoid building our own downstream images | 19:16 |
| corvus | and gear is small | 19:16 |
| mordred | ++ | 19:17 |
| fungi | we really need to reengineer the logstash stuff anyway if we're going to continue it in the long term | 19:17 |
| corvus | so if no one is violently objecting to that, i'll get something together this week | 19:17 |
| fungi | i too don't wish to heap that maintenance burden on the zuul community | 19:17 |
| mordred | corvus: clarkb was also questioning the value of our elk cluster during the ptg | 19:17 |
| mordred | corvus: ++ | 19:17 |
| fungi | well, several of us were questioning it, yes. the resources it consumes are likely disproportional with the value it provides | 19:18 |
| corvus | yeah, all the more reason to avoid over-engineering it then | 19:18 |
| ianw | do you want an action item so we come back to it next week? or just let it happen? | 19:19 |
| fungi | i'd like to revisit it next week, but that doesn't need an action item necessarily, we can just keep it on the agenda (it's relative to our config management priority effort, so already the case) | 19:20 |
| ianw | ok, the second thing i had, also probably corvus/mordred was an update on the cross-arch container work | 19:20 |
| ianw | are we at the point we have stable arm64/amd64 builds popping out? | 19:21 |
| corvus | i think that i think and i think that mordred thinks that it should all be working maybe now? | 19:21 |
| mordred | yeah - and I thnk it's time to land https://review.opendev.org/#/c/726263/ | 19:21 |
| mordred | but - it is of course possible that there is something we don't know that we don't know | 19:22 |
| corvus | but also, we sure did just take a bunch of stuff apart and put it back together, so i think we're at "lets try to use it and see if we really fixed it" | 19:22 |
| fungi | it conflicts with 726458, is that still needed? | 19:23 |
| mordred | yeah | 19:23 |
| fungi | seems to be based on an outdated patchset unless gertty is lagging for me | 19:23 |
| mordred | no - we can toss that one | 19:24 |
| ianw | i can redo https://review.opendev.org/#/c/726037/ to test it native for nodepool-builder | 19:24 |
| mordred | I went ahead and did it in the main one | 19:24 |
| fungi | okay, fine next step as far as i'm concerned then. +2 but ianw has already approved | 19:26 |
| ianw | ok, so i think wheels are in motion for that | 19:26 |
| mordred | woot | 19:26 |
| fungi | --progress-- | 19:26 |
| ianw | last one was grafana/graphite update | 19:26 |
| ianw | #link https://grafana.opendev.org/ grafana from container | 19:27 |
| ianw | that's up and production ready | 19:27 |
| ianw | graphite.opendev.org is also up, however i need to copy the data over, but i also noticed a few settings we've tweaked relating to null value storage and the mulder-scully-ish xFilesFactor that i need to port | 19:28 |
| fungi | i'll try to look at that this evening. i gather something we merged recently broke some of it | 19:28 |
| fungi | or just held up the patches to implement it? | 19:29 |
| ianw | fungi: ohh, there was deployment issues, i'd forgotten to add the promote job so it wasn't tagging the latest, but that's fixed | 19:29 |
| fungi | ahh, okay | 19:30 |
| fungi | and then yeah there were the holdups from review-test still getting deployed | 19:30 |
| fungi | with its out of control track-upstream logs | 19:30 |
| ianw | then there's been a few side issues with hosts dying and leaving hung processes, yeah the 200+ containers somehow on review-dev, and then openedge mirror disappearing | 19:30 |
| fungi | saw a few of the cron errors about the full fs | 19:30 |
| mordred | ianw: were they all manage-projects containers? | 19:31 |
| ianw | actually, it's review-test.openstack.org -- again it seems to be doing the same thing | 19:31 |
| ianw | review-test.opendev.org | 19:32 |
| mordred | I think we need to look in to our retry behavior in jeepyb - I think we currently retry indefinitely when we can't do an initial connection | 19:32 |
| mordred | although speaking of review-test ... | 19:32 |
| ianw | yeah, every hour that has launched a container | 19:32 |
| ianw | all of them looping now in " raise NoValidConnectionsError(errors)" | 19:33 |
| ianw | so, that host is on a downwards spiral again | 19:33 |
| mordred | yeah - I bet each one are spinning unable to connect to something | 19:33 |
| mordred | yup | 19:33 |
| mordred | that isn't really the behavior we want anymore | 19:34 |
| * mordred will poke at jeepyb | 19:34 | |
| mordred | also - didn't we skip review-test in manage-projects? | 19:34 |
| mordred | what is starting those? | 19:35 |
| mordred | but while we're talking about review-test - could I get some reviews on https://review.opendev.org/#/c/737023/ ? | 19:35 |
| ianw | mordred: there's a cron entry | 19:35 |
| mordred | I'd like to sync the data from review to review-test | 19:35 |
| ianw | perhaps the cron job wasn't "absented:"? | 19:35 |
| mordred | it has been noted in the past we should carefully review that - to ensure that it's not going to delete our production data. I'm pretty sure it isn't | 19:35 |
| mordred | ianw: oh - this is track-upstream isn't it? | 19:36 |
| mordred | yup. track-upstream. let's not run that on review-test :) | 19:36 |
| ianw | ok, well i also managed to kill the main container on it, so it's currently doing nothing | 19:37 |
| ianw | mordred: do you have time to subdue it back into shape, or do want me to look into it? | 19:37 |
| mordred | ianw: I'll work on it | 19:38 |
| ianw | cool, let's move on | 19:38 |
| ianw | #topic General Topics | 19:38 |
| *** openstack changes topic to "General Topics (Meeting topic: infra)" | 19:38 | |
| fungi | related, the pull-request closer is still running out of cron | 19:38 |
| fungi | d'oh, seconds too late | 19:38 |
| ianw | no we can go back if you like | 19:38 |
| fungi | nah | 19:38 |
| fungi | let's keep moving | 19:39 |
| fungi | i have corn and sausages on the grill | 19:39 |
| ianw | i think the only active trusty host transition is wiki, and i don't think there's too much happening with that ATM? | 19:39 |
| fungi | there is not, no | 19:40 |
| ianw | ok | 19:40 |
| fungi | it welcomes friends however | 19:40 |
| ianw | #topic Time to retire the openstack-infra ML yet? | 19:40 |
| *** openstack changes topic to "Time to retire the openstack-infra ML yet? (Meeting topic: infra)" | 19:40 | |
| ianw | #link http://lists.openstack.org/pipermail/openstack-infra/2020-July/006632.html | 19:40 |
| mordred | ianw: remote: https://review.opendev.org/739840 Don't install the track-upstream cron on review-test | 19:41 |
| ianw | fungi: i don't think there's been any disagreement on the list ... if anyone has an objection to why this merge should not go ahead they should speak now or forever hold your peace :) | 19:42 |
| fungi | indeed | 19:43 |
| ianw | otherwise, i guess we declare the lists joined in holy matrimony | 19:43 |
| fungi | its retirement is already on my personal schedule | 19:43 |
| corvus | no objections | 19:43 |
| ianw | mazel tov | 19:43 |
| fungi | i look forward to having one fewer list to moderate | 19:44 |
| ianw | #topic China telecom blocks | 19:44 |
| *** openstack changes topic to "China telecom blocks (Meeting topic: infra)" | 19:44 | |
| ianw | so we currently still have these in place with iptables | 19:44 |
| ianw | i think fungi has been most active in monitoring, what's the latest? | 19:44 |
| fungi | and taking the open connections graph in cacti as an indicator, the activity is ongoing from chinanet and probably elsewhere | 19:45 |
| ianw | #link http://lists.opendev.org/pipermail/service-discuss/2020-July/000053.html | 19:45 |
| ianw | that's the discussion for posterity | 19:45 |
| fungi | it stopped a lot over the weekend, but resumed late sunday utc and has been consuatnt | 19:45 |
| fungi | constant | 19:45 |
| ianw | #link https://review.opendev.org/738721 gitea reverse proxy | 19:45 |
| ianw | #link https://review.opendev.org/738725 crawler reject rules | 19:46 |
| ianw | that is, i think, the currently alternative solution, based on the fact the UA's making these requests appear to be from a very specific scraping script | 19:46 |
| fungi | basically i think we need to choose between layer 3 filtering with a lot of collateral damage, or layer 7 filtering with a lot of added complexity. rock meet hard place | 19:46 |
| corvus | i reckon we should try the layer 7 then | 19:47 |
| ianw | yeah, last week we put up the robots.txt -- but this script does not obey that | 19:47 |
| fungi | i'm personally at peace with the layer 7 solution, though it would be nice to not have to keep it in place indefinitely | 19:47 |
| corvus | btw... | 19:48 |
| corvus | how certain are we that these UA strings are non-current? | 19:48 |
| ianw | that was what i was about to say ... someone should really double check on my work there | 19:48 |
| fungi | some of that depends on how you define "current" as there is some software in use (albeit probably of questionable origin) which has decided this is the ua to report | 19:49 |
| ianw | for anyone following | 19:50 |
| ianw | #link https://review.opendev.org/#/c/738725/6/playbooks/roles/gitea/templates/gitea.vhost.j2 | 19:50 |
| ianw | is the UA's in question | 19:50 |
| fungi | if mnaser is around, i'd love to get his input. he's the first member of our community to inquire about the status, so may have additional information about the current adverse impact | 19:50 |
| ianw | the change also doesn't deploy into production, it just sets up the proxy and we still have to point haproxy at it | 19:51 |
| corvus | other than that, maybe we just land them and see if we end up getting user reports? | 19:52 |
| fungi | yeah, i'm cool approving that nowish, since it's not (yet) in a production code path | 19:52 |
| fungi | but i also second corvus's suggestion | 19:53 |
| corvus | i +3d | 19:53 |
| ianw | yeah, i guess the requests are still coming over vexxhost, but rejecting them at layer 3 v accepting them and 403'ing them probably isn't a significant networking-back-end issue? | 19:54 |
| fungi | agreed, the impact is when gitea has to process the git requests | 19:55 |
| fungi | anything which stops them before that point is probably plenty sufficient | 19:55 |
| ianw | ok, so wheels in motion there | 19:56 |
| ianw | #topic Project Renames | 19:56 |
| *** openstack changes topic to "Project Renames (Meeting topic: infra)" | 19:56 | |
| fungi | the traffic volume and request rates are not absurd on their own (though a bit aggressive and clearly ignoring modicum of a reasonable crawler) | 19:56 |
| ianw | fungi: yes ... that script that appears to be hitting us does not meet the standard of "reasonable crawler" in any way :) | 19:56 |
| ianw | there are a couple of renames, including a mis-spelling | 19:57 |
| ianw | not sure how urgent it is | 19:57 |
| ianw | and openstack/transparency-policy -> osf/transparency-policy | 19:57 |
| fungi | i'm cool with handling a rename maintenance for this week if it's warranted. i'm around and will take a closer look at the commands i cut and paste from our docs in comparison to last time | 19:58 |
| ianw | i'm out thu/fri .au time so no help | 19:58 |
| fungi | the transparency-policy move is definitely non-urgent. that repo is basically dead since years | 19:58 |
| ianw | perhaps we wait then | 19:59 |
| fungi | the topiko->tobiko fix is the only one i expect has much urgency behind it | 19:59 |
| fungi | but... folks involved signed off on the rename patch which incorporated that typo, so i'm not in any hurry either | 20:00 |
| fungi | they seem to be patient | 20:00 |
| ianw | ok, we can come back to it next week | 20:00 |
| ianw | #topic Open Discussion | 20:00 |
| *** openstack changes topic to "Open Discussion (Meeting topic: infra)" | 20:00 | |
| ianw | we are at time, but if anyone would like to flag something i guess we can go a little over | 20:01 |
| fungi | i'm good. stuff's about ready to come off the grill anyway | 20:01 |
| fungi | and i need to stop obsessing over the bouncy castle next door | 20:01 |
| ianw | we've unfortunately gone back into lockdown here :( | 20:02 |
| ianw | so my daughter will have to cancel her bday party since we can't have anyone over | 20:02 |
| ianw | it is certainly a year to remember | 20:03 |
| ianw | with that, i think we're done, thanks everyone | 20:03 |
| ianw | #endmeeting | 20:03 |
| *** openstack changes topic to "Incident management and meetings for the OpenDev sysadmins; normal discussions are in #opendev" | 20:03 | |
| openstack | Meeting ended Tue Jul 7 20:03:31 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 20:03 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-07-07-19.00.html | 20:03 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-07-07-19.00.txt | 20:03 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-07-07-19.00.log.html | 20:03 |
| fungi | thanks ianw! | 20:03 |
| *** tobiash has quit IRC | 22:12 | |
| *** jentoio has quit IRC | 22:43 | |
| *** jentoio has joined #opendev-meeting | 22:45 | |
| *** mnaser has quit IRC | 22:50 | |
| *** jentoio has quit IRC | 22:52 | |
| *** zbr has quit IRC | 22:53 | |
| *** diablo_rojo_phon has quit IRC | 22:53 | |
| *** hamalq_ has quit IRC | 23:10 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!