| *** hamalq has quit IRC | 00:58 | |
| *** hamalq has joined #opendev-meeting | 01:14 | |
| *** hamalq has quit IRC | 01:31 | |
| *** hamalq has joined #opendev-meeting | 01:37 | |
| *** hamalq has quit IRC | 02:32 | |
| *** hamalq has joined #opendev-meeting | 02:57 | |
| *** hamalq has quit IRC | 06:10 | |
| *** hamalq has joined #opendev-meeting | 06:12 | |
| *** hamalq has quit IRC | 06:59 | |
| *** diablo_rojo has quit IRC | 09:54 | |
| jentoio | #startmeeting OPNFV TSC | 12:59 |
|---|---|---|
| openstack | Meeting started Tue Jun 16 12:59:52 2020 UTC and is due to finish in 60 minutes. The chair is jentoio. Information about MeetBot at http://wiki.debian.org/MeetBot. | 12:59 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 12:59 |
| *** openstack changes topic to " (Meeting topic: OPNFV TSC)" | 12:59 | |
| openstack | The meeting name has been set to 'opnfv_tsc' | 12:59 |
| AJaeger | jentoio: you're in the wrong channel! | 13:00 |
| jentoio | yup, tsorry ;( | 13:00 |
| AJaeger | please issue an #endmeeting here | 13:00 |
| jentoio | #endmeeting | 13:01 |
| *** openstack changes topic to "Incident management and meetings for the OpenDev sysadmins; normal discussions are in #opendev" | 13:01 | |
| openstack | Meeting ended Tue Jun 16 13:01:01 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 13:01 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/opnfv_tsc/2020/opnfv_tsc.2020-06-16-12.59.html | 13:01 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/opnfv_tsc/2020/opnfv_tsc.2020-06-16-12.59.txt | 13:01 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/opnfv_tsc/2020/opnfv_tsc.2020-06-16-12.59.log.html | 13:01 |
| *** mnaser has quit IRC | 14:14 | |
| *** zbr_ has joined #opendev-meeting | 14:15 | |
| *** zbr has quit IRC | 14:16 | |
| *** zbr_ is now known as zbr | 14:16 | |
| *** mnaser has joined #opendev-meeting | 14:17 | |
| *** hamalq has joined #opendev-meeting | 16:06 | |
| *** diablo_rojo has joined #opendev-meeting | 16:27 | |
| -openstackstatus- NOTICE: Zuul is being restarted for an urgent configuration change and may be offline for 15-30 minutes. Patches uploaded or approved during that time will need to be rechecked. | 16:37 | |
| *** hamalq_ has joined #opendev-meeting | 16:56 | |
| *** hamalq has quit IRC | 16:59 | |
| -openstackstatus- NOTICE: Zuul is back online; changes uploaded or approved between 16:40 and 18:00 will need to be rechecked. | 18:06 | |
| *** hamalq_ has quit IRC | 18:17 | |
| *** hamalq has joined #opendev-meeting | 18:18 | |
| clarkb | Anyone else here for the meeting? we'll get started in a couple minuts | 18:59 |
| diablo_rojo | o/ | 18:59 |
| ianw | o/ | 19:00 |
| clarkb | #startmeeting infra | 19:01 |
| openstack | Meeting started Tue Jun 16 19:01:06 2020 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:01 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:01 |
| *** openstack changes topic to " (Meeting topic: infra)" | 19:01 | |
| openstack | The meeting name has been set to 'infra' | 19:01 |
| clarkb | #link http://lists.opendev.org/pipermail/service-discuss/2020-June/000039.html Our Agenda | 19:01 |
| clarkb | #topic Announcements | 19:01 |
| *** openstack changes topic to "Announcements (Meeting topic: infra)" | 19:01 | |
| clarkb | I didn't have any announcements | 19:01 |
| clarkb | #topic Actions from last meeting | 19:02 |
| *** openstack changes topic to "Actions from last meeting (Meeting topic: infra)" | 19:02 | |
| clarkb | #link http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-06-09-19.01.txt minutes from last meeting | 19:02 |
| clarkb | no actions recorded, but it is feeling like things are returning to normal after the PTG. Oddly ti seemed like we still had the quiet week last week even though people didn't need to travel | 19:02 |
| clarkb | (maybe that was just my perception) | 19:02 |
| clarkb | #topic Specs approval | 19:02 |
| *** openstack changes topic to "Specs approval (Meeting topic: infra)" | 19:02 | |
| clarkb | #link https://review.opendev.org/#/c/731838/ Authentication broker service | 19:02 |
| clarkb | This isn't ready for approval yet, but wanted to keep pointing eyeballs towards it | 19:03 |
| mordred | o/ | 19:03 |
| clarkb | fungi: ^ anything else to say about this spec? | 19:03 |
| fungi | i haven't updated it yet, had a "quiet week" ;) | 19:03 |
| fungi | more comments appreciated though | 19:03 |
| corvus | o/ | 19:03 |
| * mordred looks forward to fungi's updates | 19:03 | |
| clarkb | #topic Priority Efforts | 19:04 |
| *** openstack changes topic to "Priority Efforts (Meeting topic: infra)" | 19:04 | |
| clarkb | #topic Update Config Management | 19:05 |
| *** openstack changes topic to "Update Config Management (Meeting topic: infra)" | 19:05 | |
| clarkb | I'm not aware of a ton of changes here recently. Anyone have topics to bring up? | 19:05 |
| mordred | corvus improved the disable-ansible script | 19:06 |
| clarkb | (I'm moving somewhat quickly beacuse our general topics list is pretty long this week and want to be sure we get there but feel free to bring stuff up under priority topics if relevant) | 19:06 |
| mordred | nope. /me shuts up | 19:06 |
| fungi | i've got a half-baked change to move the rest of our repo mirrornig configuration from puppet to ansible | 19:06 |
| clarkb | mordred: thats a good call out. We should get into the habit of providing detailed reasons for disabling ansible there | 19:06 |
| mordred | \o/ | 19:06 |
| clarkb | fungi: is that ready for review yet? | 19:07 |
| fungi | clarkb: it's ready for suggestions, but no it's not ready to merge yet | 19:07 |
| fungi | it's a lot of me learning some ansible and jinja concepts for the first time | 19:07 |
| clarkb | #link https://review.opendev.org/#/c/735406/ Ansiblify reprepro configs. Is WIP comments welcome | 19:07 |
| fungi | it will be a pretty massive diff | 19:08 |
| fungi | (once complete) | 19:08 |
| clarkb | it should be a pretty safe transition too as we can avoid releasing volumes until we are happy with the end results? | 19:09 |
| clarkb | thanks for working on that | 19:09 |
| fungi | yep, once i rework it following your earlier suggestion | 19:09 |
| clarkb | #topic OpenDev | 19:10 |
| *** openstack changes topic to "OpenDev (Meeting topic: infra)" | 19:10 | |
| clarkb | I don't have much to add here. I've completely failed at sending reminder emails about the advisory board but mnaser has responded. Thank you! | 19:10 |
| clarkb | I'll really try to get to that this week | 19:10 |
| mnaser | \o/ | 19:10 |
| * mordred hands mnaser an orange | 19:12 | |
| clarkb | Anything else to add re OpenDev? | 19:12 |
| mordred | oh - | 19:13 |
| mordred | this is really minor - but I snagged the opendev freenode nick yesterday and put it in our secrets file (thanks for the suggestion mnaser) | 19:13 |
| mordred | in case we want to use it for opendev-branded bots | 19:14 |
| clarkb | thanks | 19:14 |
| clarkb | #topic General Topics | 19:15 |
| *** openstack changes topic to "General Topics (Meeting topic: infra)" | 19:15 | |
| clarkb | #topic pip-and-virtualenv | 19:15 |
| *** openstack changes topic to "pip-and-virtualenv (Meeting topic: infra)" | 19:15 | |
| clarkb | This change has landed and we're starting to see more and more fallout from it. Nothing unexpected yet I don't think | 19:16 |
| clarkb | possibly even a new case of it in #zuul right now too :) | 19:16 |
| clarkb | Keep an eye out for problems and thank you to AJaeger and mordred for jumping on fixes | 19:16 |
| clarkb | ianw: where are we with considering the spec completed? and cleanup in the nodepool configs? can we start on that or should we wait a bit logner? | 19:17 |
| ianw | i have pushed changes to do cleanup of the old jobs and images | 19:17 |
| ianw | i guess it's not too high a priority right now, the changes are there and i'll keep on them as appropriate | 19:18 |
| clarkb | thanks. Should I push up a change to mark the spec compelted? or wait a bit more on cleanup for that? | 19:19 |
| ianw | i guess we can mark it complete, if we consider dealing with the fallout complete :) | 19:20 |
| ianw | i thought it was too quiet yesterday, so i'll try to catch up on anything i've missed | 19:20 |
| clarkb | ianw: I think a lot of people may have had friday and monday off or something | 19:21 |
| clarkb | because ya definitely seems to be picking up now | 19:21 |
| corvus | 40% of vacation days are fridays or mondays | 19:21 |
| clarkb | #topic Zookeeper TLS | 19:23 |
| *** openstack changes topic to "Zookeeper TLS (Meeting topic: infra)" | 19:23 | |
| clarkb | This is the thing that led to the ansible limbo | 19:23 |
| clarkb | corvus: want to walk us through this? | 19:23 |
| corvus | this topic was *supposed* to be about scheduling some downtime for friday to switch to zk tls | 19:23 |
| corvus | but as it turns out, this morning we switched to tls and switched back already | 19:23 |
| corvus | the short version is that yesterday our self-signed gearman certs expired | 19:24 |
| corvus | well, technically just the ca cert | 19:24 |
| corvus | which means that no zuul component could connect to gearman. so we lost the use of the zuul cli, and if any component were restarted for any reason, it would be unable to connect, so the system would decay | 19:25 |
| corvus | correcting that required a full restart, as did the zk tls work, so we decided to combine them | 19:25 |
| corvus | unfortunately, shortly after starting the nodepool launchers, we ran into a bug | 19:25 |
| corvus | #link kazoo bug https://github.com/python-zk/kazoo/issues/587 | 19:25 |
| corvus | so we manually reverted the tls change (leaving the new gear certs in place) | 19:26 |
| corvus | and everything is running again. | 19:26 |
| corvus | next steps: make sure this is merged: | 19:26 |
| corvus | #link revert zk tls https://review.opendev.org/735990 | 19:26 |
| corvus | then when it is, we can clear the disable-ansible file and resume speed | 19:27 |
| corvus | after that, i'm going to look into running zk in a mode where it can accept tls and plain connections | 19:27 |
| clarkb | and once disable ansible is cleared we'll get updates to a few docker images, apply dns zone backups, and change rsync flags for centos/fedora mirrors | 19:27 |
| corvus | if that's possible, i'd like to restart the zk cluster with that, and then try to repro the bug against production | 19:27 |
| clarkb | calling that out so toehrs are aware there will be a few changes landing once ansible is reenabled | 19:27 |
| clarkb | corvus: ++ I like that plan | 19:28 |
| mordred | ++ | 19:28 |
| corvus | based on info from tobiash, we suspect it may have to do with response size, so it may help to get a repro case out of production data | 19:28 |
| clarkb | corvus: we should be able to easily switch over a single builder or launcher without major impact to production to help sort out what is going on | 19:28 |
| corvus | clarkb: agreed | 19:28 |
| corvus | eot | 19:29 |
| clarkb | #topic DNS Cleanup | 19:29 |
| *** openstack changes topic to "DNS Cleanup (Meeting topic: infra)" | 19:29 | |
| clarkb | The change to implement the recording of zone data has landed and should apply to bridge when ansibel starts rerunning | 19:30 |
| clarkb | ianw: ^ fyi | 19:30 |
| clarkb | probably want to make sure that is working properly once it goes in? | 19:30 |
| ianw | yeah, i added the credentials so should be ok | 19:30 |
| clarkb | I annotated the etherpad with notes on things I thinkwe can cleanup | 19:31 |
| clarkb | what are we thinking about for cleanup? wait for backups to run with existing records first so we've got that info recorded then do cleanup? | 19:31 |
| clarkb | (that is sort of what I thought would be a good process) | 19:31 |
| mordred | yeah | 19:31 |
| clarkb | k I can help with the button clicking to clean up records once we're at that point | 19:32 |
| clarkb | ianw: anything else worth mentioning on this topic? | 19:32 |
| ianw | yeah, we can iterate on it a bit then too, as the list gets shorter it's easier to see what can go :) | 19:32 |
| ianw | nope, thanks | 19:32 |
| clarkb | sounds good, thanks for putting this together | 19:32 |
| clarkb | #topic Etherpad Upgrade to 1.8.4 or 1.8.5 | 19:32 |
| *** openstack changes topic to "Etherpad Upgrade to 1.8.4 or 1.8.5 (Meeting topic: infra)" | 19:32 | |
| clarkb | Fungi did some work to get our etherpad server upgraded to 1.8.4 (from 1.8.0) | 19:33 |
| clarkb | we then noticed that there was a UI rendering bug when testing that | 19:33 |
| clarkb | #link https://review.opendev.org/#/c/729029/ Upgrade Etherpad | 19:33 |
| fungi | not dissimilar from some of the weirdness we noticed with author colors when going through meetpad | 19:33 |
| clarkb | this change now addresses that with a simple css hack that I came up with. Upstream they've fixed this differently with a fairly large css refactor and we should see that in the next release (1.8.5?( | 19:33 |
| fungi | i wonder if that will also resolve it for meetpad uses | 19:34 |
| clarkb | the question I've got is do we think we should upgrade with the workaround as 1.8.4 includes a bug fix around db writes? or wait for 1.8.5 to avoid potential UI weirdness | 19:34 |
| fungi | i'm in no huge hurry. i'm excited for the potential fix for perpetually "loading..." pads, but other than that there's no urgency | 19:35 |
| mordred | there's no urgency, but the workaround isn't super onerous either | 19:35 |
| mordred | so I'm ok rolling it out in that form - or with waiting | 19:35 |
| mordred | we can remove the sed from our dockerfile when we bump the version | 19:35 |
| mordred | (it's not like one of those "in a few months we're not going to be paying attention and our local hack is going to bork us) | 19:36 |
| corvus | clarkb: what's the workaround? | 19:36 |
| corvus | oh i see it now | 19:36 |
| corvus | sorry, buried in the run cmd | 19:36 |
| clarkb | corvus: changing the padding between the spans that contain lines | 19:36 |
| clarkb | the way the padding was set up before caused the lines to overlap so their colors successively covered each other | 19:37 |
| corvus | i agree it seems safe to move forward with 029 | 19:37 |
| fungi | you tracked that back to a particular change between 1.8.1 and 1.8.3 yeah? | 19:37 |
| clarkb | the bug is also purely cosmetic so shouldn't affect content directly, just how we see it | 19:38 |
| clarkb | fungi: ya its in 1.8.3 (there was no 1.8.1 or 1.8.2 iirc) | 19:38 |
| fungi | yeah, there was at least no 1.8.2, for sure | 19:39 |
| clarkb | I think what I'm taking away from this is if everything else calms down (uwsgi, pip/virtualenv, dns, zk tls, etc) then we can go ahead with this and watch it | 19:39 |
| fungi | sounds fine to me | 19:39 |
| clarkb | thanks for the feedback | 19:39 |
| clarkb | and if 1.8.5 happens before then we can drop the workaround | 19:40 |
| clarkb | #topic Getting more stuff off of python2 | 19:40 |
| *** openstack changes topic to "Getting more stuff off of python2 (Meeting topic: infra)" | 19:40 | |
| clarkb | One of the things that came out of the PTG was it would be useful for those a bit more familiar with our systems to do an audit of where we stand with python2 usage. This way others can dive in and port or switch runtimes | 19:41 |
| clarkb | #link https://etherpad.opendev.org/p/opendev-tools-still-running-python2 | 19:41 |
| clarkb | I've started this audit in that etherpad. It is nowhere near complete. One thing that I have discovered is that a lot of our software is python3 capable but running under python2. We'll want to keep that in mind as we update configuration management that a good next step is to switch the runtime too | 19:42 |
| clarkb | I have also found a couple cases that are definitely python2 only right now. Our use of supybot for meetbot and the logstash data pipeline. For meetbot we have a spec to replace it already which is noted on the etherpad | 19:42 |
| clarkb | if you can think of other tools we need to be checking on feel free to add them to the list and I can dig in further as I have time | 19:42 |
| clarkb | The goal here isn't really to fix everything as much as to be aware of what needs fixing | 19:43 |
| fungi | as we move services and automation to platforms without python 2.7, we can fix things where needed | 19:45 |
| fungi | if they don't become urgent before that | 19:45 |
| clarkb | yup and it gives people a list of things they can pick off over time if they want to help out | 19:45 |
| mordred | yeah - most of the things are easy enough to work on - but are pretty opaque that they need to be worked on | 19:45 |
| mordred | what clarkb said | 19:45 |
| fungi | useful to know where we expect the pain points for such moves to be though | 19:45 |
| clarkb | #topic Trusty Upgrades | 19:46 |
| *** openstack changes topic to "Trusty Upgrades (Meeting topic: infra)" | 19:46 | |
| clarkb | I don't have much to add on this topic but did want to point out it seems that osf's interop working group is picking up some steam. I'm hoping that may translate into some better interest/support for refstack | 19:47 |
| clarkb | and we can maybe channel that into a refstack server upgrade. The docker work I did is actually really close. Its mostly a matter of having someone test it now (which I'm hoping the interop wg can do) | 19:47 |
| clarkb | #topic Open Discussion | 19:48 |
| *** openstack changes topic to "Open Discussion (Meeting topic: infra)" | 19:48 | |
| clarkb | Anything else to bring up before we end the meeting? | 19:49 |
| fungi | it's just come to light in #opendev that the zuul restart switched our ansible default from 2.8 to 2.9, so expect possible additional behavior changes | 19:51 |
| clarkb | fungi: have we confirmed that (I theorized it and wouldn't be surprised) | 19:51 |
| fungi | ianw: compared logs which showed passing results used 2.8 and the failure observed used 2.9 | 19:51 |
| corvus | yes, latest zuul master uses 2.9 by default | 19:53 |
| corvus | we can pin the openstack tenant to 2.8 if we need to | 19:53 |
| clarkb | so far its only popped up as a single issue which has a fix | 19:53 |
| clarkb | I guess if it gets worse we can pin | 19:53 |
| fungi | i don't know yet if that's warranted, may be the issues are small enough we can just fix them | 19:53 |
| corvus | though istr we did some testing around this and didn't see a lot of issues | 19:53 |
| fungi | apparently match as a filter is one | 19:54 |
| clarkb | corvus: ya we tested with a lot of general playbooks in zuul-jobs | 19:54 |
| corvus | so yeah, i think it's probably best to roll forward | 19:54 |
| frickler | also devstack is broken by a new uwsgi release and still needs fixes | 19:54 |
| frickler | and neutron-grenade-multinode seems to suffer from the venv removal | 19:54 |
| clarkb | frickler: ya I meant to look into why multinode was sad about that | 19:55 |
| clarkb | the last day and a half have been distracting :) After lunch today I'll try to be useful for all the things | 19:56 |
| clarkb | sounds like that may be it for our meeting | 19:58 |
| clarkb | thank you everyone! | 19:58 |
| clarkb | #endmeeting | 19:58 |
| *** openstack changes topic to "Incident management and meetings for the OpenDev sysadmins; normal discussions are in #opendev" | 19:58 | |
| openstack | Meeting ended Tue Jun 16 19:58:21 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 19:58 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-06-16-19.01.html | 19:58 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-06-16-19.01.txt | 19:58 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-06-16-19.01.log.html | 19:58 |
| fungi | thanks clarkb! | 19:58 |
| *** yoctozepto has quit IRC | 22:33 | |
| *** corvus has quit IRC | 22:33 | |
| *** tristanC has quit IRC | 22:33 | |
| *** SotK has quit IRC | 22:33 | |
| *** tobiash has quit IRC | 22:33 | |
| *** mnaser has quit IRC | 22:33 | |
| *** hamalq has quit IRC | 22:33 | |
| *** AJaeger has quit IRC | 22:33 | |
| *** ChanServ has quit IRC | 22:33 | |
| *** ianw has quit IRC | 22:33 | |
| *** frickler has quit IRC | 22:33 | |
| *** zbr has quit IRC | 22:33 | |
| *** mordred has quit IRC | 22:33 | |
| *** fungi has quit IRC | 22:33 | |
| *** diablo_rojo has quit IRC | 22:33 | |
| *** clarkb has quit IRC | 22:33 | |
| *** tristanC has joined #opendev-meeting | 22:37 | |
| *** AJaeger has joined #opendev-meeting | 22:37 | |
| *** fungi has joined #opendev-meeting | 22:37 | |
| *** mordred has joined #opendev-meeting | 22:37 | |
| *** zbr has joined #opendev-meeting | 22:37 | |
| *** hamalq has joined #opendev-meeting | 22:37 | |
| *** tobiash has joined #opendev-meeting | 22:37 | |
| *** SotK has joined #opendev-meeting | 22:37 | |
| *** ChanServ has joined #opendev-meeting | 22:37 | |
| *** corvus has joined #opendev-meeting | 22:37 | |
| *** tepper.freenode.net sets mode: +o ChanServ | 22:37 | |
| *** yoctozepto has joined #opendev-meeting | 22:37 | |
| *** ianw has joined #opendev-meeting | 22:38 | |
| *** frickler has joined #opendev-meeting | 22:38 | |
| *** mnaser has joined #opendev-meeting | 22:39 | |
| *** clarkb has joined #opendev-meeting | 22:39 | |
| *** mnaser has quit IRC | 22:41 | |
| *** mnaser has joined #opendev-meeting | 22:48 | |
| *** diablo_rojo has joined #opendev-meeting | 22:52 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!