| jamesmcarthur | efried: I'm here! Obviously too late :) | 00:03 |
|---|---|---|
| *** wolverineav has joined #openstack-infra | 00:07 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Add API endpoint to get frozen jobs https://review.openstack.org/607077 | 00:07 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Get executor job params https://review.openstack.org/607078 | 00:07 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Separate out executor server from runner https://review.openstack.org/607079 | 00:10 |
| *** armax has joined #openstack-infra | 00:10 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: runner: implement prep-workspace https://review.openstack.org/607082 | 00:11 |
| *** gyee has quit IRC | 00:11 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: runner: add configuration schema https://review.openstack.org/640672 | 00:11 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: runner: add execute sub-command https://review.openstack.org/630944 | 00:11 |
| *** mriedem_afk is now known as mriedem | 00:12 | |
| *** irclogbot_1 has joined #openstack-infra | 00:17 | |
| ianw | are the deb-* and fuel-* repos actually worth updating? | 00:21 |
| clarkb | ianw I think you can skip any that are retired in projects.yaml. I know all the deb-* repos meet that criteria | 00:22 |
| *** wolverineav has quit IRC | 00:23 | |
| ianw | great point ... | 00:26 |
| *** rascasoft has joined #openstack-infra | 00:28 | |
| *** jamesmcarthur has quit IRC | 00:30 | |
| *** rascasoft has quit IRC | 00:36 | |
| *** armax has quit IRC | 00:44 | |
| *** ricolin has joined #openstack-infra | 01:05 | |
| *** rascasoft has joined #openstack-infra | 01:45 | |
| *** jamesmcarthur has joined #openstack-infra | 01:46 | |
| *** rascasoft has quit IRC | 01:54 | |
| *** Sundar has quit IRC | 02:02 | |
| *** jamesmcarthur has quit IRC | 02:03 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Replace openstack.org git:// URLs with https:// https://review.openstack.org/645440 | 02:21 |
| *** hongbin has joined #openstack-infra | 02:22 | |
| *** wolverineav has joined #openstack-infra | 02:24 | |
| *** diablo_rojo has quit IRC | 02:25 | |
| *** wolverineav has quit IRC | 02:28 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/devstack-gate master: Replace openstack.org git:// URLs with https:// https://review.openstack.org/645451 | 02:39 |
| *** psachin has joined #openstack-infra | 02:48 | |
| *** yamamoto has joined #openstack-infra | 02:49 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Replace openstack.org git:// URLs with https:// https://review.openstack.org/645440 | 02:50 |
| *** rascasoft has joined #openstack-infra | 03:00 | |
| mriedem | ianw: gmann: can we land https://review.openstack.org/#/c/644638/ ? i just hit it again in another place (keystone failed to import memcache) | 03:04 |
| ianw | mridem: i'm ok if you want to merge it ... i just wasn't like 100% confident I'd followed every nuance ... | 03:07 |
| mriedem | yeah it's rare b/c only one node provider has preloaded pips it looks like - rax-dfw | 03:08 |
| mriedem | and some of these packages don't have 3.6 classifiers | 03:08 |
| mriedem | apparently memcached is another one | 03:08 |
| ianw | yeah ... that ... just shouldn't be the case :/ | 03:09 |
| *** rascasoft has quit IRC | 03:10 | |
| *** apetrich has quit IRC | 03:15 | |
| *** udesale has joined #openstack-infra | 03:21 | |
| gmann | mriedem: ianw +A | 03:26 |
| mriedem | thanks | 03:26 |
| *** mriedem has quit IRC | 03:29 | |
| *** ramishra has joined #openstack-infra | 03:38 | |
| *** whoami-rajat has joined #openstack-infra | 03:41 | |
| *** yamamoto has quit IRC | 03:50 | |
| *** jamesmcarthur has joined #openstack-infra | 03:51 | |
| *** lseki has quit IRC | 03:52 | |
| *** nicolasbock has quit IRC | 04:03 | |
| *** jamesmcarthur has quit IRC | 04:12 | |
| *** hongbin has quit IRC | 04:12 | |
| *** ykarel has joined #openstack-infra | 04:16 | |
| *** yamamoto has joined #openstack-infra | 04:25 | |
| *** hamzy_ has joined #openstack-infra | 04:31 | |
| *** TheJulia has quit IRC | 04:31 | |
| *** coreycb has quit IRC | 04:31 | |
| *** portdirect has quit IRC | 04:31 | |
| *** TheJulia has joined #openstack-infra | 04:31 | |
| *** dustinc has quit IRC | 04:31 | |
| *** spsurya has quit IRC | 04:31 | |
| *** PrinzElvis has quit IRC | 04:31 | |
| *** evgenyl has quit IRC | 04:32 | |
| *** adrianreza has quit IRC | 04:32 | |
| *** sparkycollier has quit IRC | 04:32 | |
| *** zaro has quit IRC | 04:32 | |
| *** kmalloc has quit IRC | 04:32 | |
| *** hogepodge has quit IRC | 04:32 | |
| *** srwilkers has quit IRC | 04:32 | |
| *** johnsom has quit IRC | 04:32 | |
| *** Ng has quit IRC | 04:33 | |
| *** jbryce has quit IRC | 04:33 | |
| *** hamzy has quit IRC | 04:33 | |
| *** chrisyang_0660 has quit IRC | 04:33 | |
| *** evgenyl has joined #openstack-infra | 04:34 | |
| *** srwilkers has joined #openstack-infra | 04:34 | |
| *** sparkycollier has joined #openstack-infra | 04:34 | |
| *** PrinzElvis has joined #openstack-infra | 04:34 | |
| *** portdirect has joined #openstack-infra | 04:34 | |
| *** adrianreza has joined #openstack-infra | 04:34 | |
| *** coreycb has joined #openstack-infra | 04:34 | |
| *** spsurya has joined #openstack-infra | 04:34 | |
| *** johnsom has joined #openstack-infra | 04:34 | |
| *** hogepodge has joined #openstack-infra | 04:35 | |
| *** chrisyang_0660 has joined #openstack-infra | 04:36 | |
| *** dayou has quit IRC | 04:36 | |
| *** jbryce has joined #openstack-infra | 04:38 | |
| *** dustinc has joined #openstack-infra | 04:39 | |
| *** Ng has joined #openstack-infra | 04:39 | |
| *** zaro has joined #openstack-infra | 04:39 | |
| *** kmalloc has joined #openstack-infra | 04:39 | |
| *** udesale has quit IRC | 04:47 | |
| *** udesale has joined #openstack-infra | 04:47 | |
| *** jamesmcarthur has joined #openstack-infra | 04:52 | |
| *** lpetrut has joined #openstack-infra | 04:52 | |
| *** jamesmcarthur has quit IRC | 04:57 | |
| *** janki has joined #openstack-infra | 05:04 | |
| *** raukadah is now known as chandankumar | 05:06 | |
| *** rascasoft has joined #openstack-infra | 05:17 | |
| *** lpetrut has quit IRC | 05:25 | |
| *** rascasoft has quit IRC | 05:28 | |
| *** dustinc has quit IRC | 05:50 | |
| *** tkajinam has quit IRC | 05:59 | |
| *** tkajinam has joined #openstack-infra | 05:59 | |
| *** lpetrut has joined #openstack-infra | 06:17 | |
| *** jtomasek has joined #openstack-infra | 06:17 | |
| *** psachin has quit IRC | 06:20 | |
| *** wolverineav has joined #openstack-infra | 06:24 | |
| *** psachin has joined #openstack-infra | 06:25 | |
| *** david-lyle has joined #openstack-infra | 06:27 | |
| *** dklyle has quit IRC | 06:27 | |
| *** rascasoft has joined #openstack-infra | 06:29 | |
| *** david-lyle has quit IRC | 06:29 | |
| *** dklyle has joined #openstack-infra | 06:29 | |
| *** wolverineav has quit IRC | 06:29 | |
| *** lpetrut has quit IRC | 06:30 | |
| *** rascasoft has quit IRC | 06:33 | |
| *** kjackal has joined #openstack-infra | 06:35 | |
| *** dims has quit IRC | 06:39 | |
| *** dims has joined #openstack-infra | 06:41 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [dmn] Stashing some scripts to make git:// -> https:// changes https://review.openstack.org/642314 | 07:00 |
| ianw | infra-root: ^ given those a pretty good workout, including posting the change, adding a message about auto-merging, and auto-merging some devstack ones to test (https://review.openstack.org/#/q/status:merged+topic:opendev-gerrit-git) | 07:01 |
| jhesketh | ianw: I'm very late to the party, but did you see my email on the discuss list? | 07:03 |
| *** rascasoft has joined #openstack-infra | 07:03 | |
| ianw | jhesketh: when was it sent ;) looking now | 07:03 |
| jhesketh | this afternoon I think | 07:03 |
| jhesketh | re [infra][dev] Options for upcoming git:// to https:// transition | 07:04 |
| ianw | jhesketh: oh, yeah adding in renames was pretty much option 3 of the original mail? | 07:04 |
| *** pgaxatte has joined #openstack-infra | 07:05 | |
| jhesketh | ianw: ah, I took option #3 as doing it during part of the migration/cut over. My suggestion was to do the rename as a mass set of proposed changes now (ie earlier than things like gerrit moving etc) | 07:06 |
| *** harlowja has quit IRC | 07:06 | |
| *** rcernin has quit IRC | 07:07 | |
| ianw | jhesketh: but we haven't finalised if it will be git.opendev.org/openstack/nova or git.opendev.org/nova/nova right? so that fixup needs to happen *after* all that is sorted? | 07:07 |
| ianw | whereas, git.openstack.org/openstack/nova will be redirected to the right place (via a lot of modrewrite magic ultimately) no matter where it ends up in opendev.org | 07:08 |
| ianw | I should say "https://git.openstack.org/openstack/nova", to be clear | 07:08 |
| jhesketh | hmm okay, that's fair | 07:09 |
| jhesketh | I'm happy with option 1 FWIW, was mostly thinking out loud | 07:10 |
| jhesketh | (but I've also obviously been a little disconnected from the migration too which is entirely my fault) | 07:10 |
| ianw | np ... plenty of ideas to go around with this transition! :) | 07:10 |
| *** gfidente has joined #openstack-infra | 07:10 | |
| *** whoami-rajat has quit IRC | 07:11 | |
| *** lpetrut has joined #openstack-infra | 07:11 | |
| ianw | i'll respond on list just to close the loop | 07:13 |
| jhesketh | sure :-) | 07:14 |
| *** dpawlik_ is now known as dpawlik | 07:21 | |
| *** kjackal has quit IRC | 07:22 | |
| *** kjackal has joined #openstack-infra | 07:22 | |
| *** pcaruana has joined #openstack-infra | 07:27 | |
| *** whoami-rajat has joined #openstack-infra | 07:30 | |
| *** ramishra has quit IRC | 07:32 | |
| *** rpittau|afk is now known as rpittau | 07:33 | |
| *** kopecmartin|off is now known as kopecmartin | 07:37 | |
| *** ykarel_ has joined #openstack-infra | 07:37 | |
| *** ykarel_ has quit IRC | 07:37 | |
| *** ykarel_ has joined #openstack-infra | 07:39 | |
| *** ykarel_ has quit IRC | 07:39 | |
| *** ykarel has quit IRC | 07:40 | |
| *** apetrich has joined #openstack-infra | 07:43 | |
| *** ykarel has joined #openstack-infra | 07:47 | |
| *** tosky has joined #openstack-infra | 07:53 | |
| *** ramishra has joined #openstack-infra | 07:56 | |
| *** yamamoto has quit IRC | 08:02 | |
| *** yamamoto has joined #openstack-infra | 08:03 | |
| *** lpetrut has quit IRC | 08:04 | |
| *** xek_ has joined #openstack-infra | 08:08 | |
| *** ginopc has joined #openstack-infra | 08:10 | |
| *** helenaAM has joined #openstack-infra | 08:27 | |
| *** iurygregory has joined #openstack-infra | 08:32 | |
| *** dtantsur|afk is now known as dtantsur | 08:33 | |
| *** tkajinam has quit IRC | 08:34 | |
| *** jpich has joined #openstack-infra | 08:43 | |
| *** ykarel is now known as ykarel|lunch | 08:48 | |
| *** jpena|off is now known as jpena | 08:51 | |
| *** jbadiapa has joined #openstack-infra | 09:02 | |
| *** tobias-urdin has joined #openstack-infra | 09:15 | |
| *** ricolin has quit IRC | 09:22 | |
| *** kjackal has quit IRC | 09:22 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Update docs for provider removal. https://review.openstack.org/645220 | 09:27 |
| *** derekh has joined #openstack-infra | 09:42 | |
| *** kjackal has joined #openstack-infra | 09:43 | |
| *** dayou has joined #openstack-infra | 09:43 | |
| *** Lucas_Gray has joined #openstack-infra | 09:50 | |
| *** roman_g has joined #openstack-infra | 09:58 | |
| *** jbadiapa has quit IRC | 10:00 | |
| *** lpetrut has joined #openstack-infra | 10:04 | |
| *** ykarel|lunch is now known as ykarel | 10:08 | |
| *** ramishra_ has joined #openstack-infra | 10:10 | |
| *** ramishra has quit IRC | 10:12 | |
| *** jpich has quit IRC | 10:13 | |
| *** jpich has joined #openstack-infra | 10:14 | |
| *** jbadiapa has joined #openstack-infra | 10:19 | |
| *** lpetrut has quit IRC | 10:23 | |
| *** dtantsur is now known as dtantsur|brb | 10:28 | |
| *** nicolasbock has joined #openstack-infra | 10:40 | |
| *** rascasoft has quit IRC | 10:42 | |
| openstackgerrit | Luigi Toscano proposed openstack-infra/zuul-jobs master: DNM Debug stage-output, change archival mechanism https://review.openstack.org/645239 | 10:42 |
| *** rascasoft has joined #openstack-infra | 10:43 | |
| *** kopecmartin is now known as kopecmartin|lunc | 10:47 | |
| *** chrisyang_0660 has quit IRC | 10:59 | |
| *** johnsom has quit IRC | 10:59 | |
| *** chrisyang_0660 has joined #openstack-infra | 10:59 | |
| *** johnsom has joined #openstack-infra | 10:59 | |
| *** adrianreza has quit IRC | 10:59 | |
| *** sparkycollier has quit IRC | 10:59 | |
| *** whoami-rajat has quit IRC | 10:59 | |
| *** spsurya has quit IRC | 10:59 | |
| *** portdirect has quit IRC | 11:00 | |
| *** sparkycollier has joined #openstack-infra | 11:00 | |
| *** dougwig has quit IRC | 11:00 | |
| *** adrianreza has joined #openstack-infra | 11:00 | |
| *** wolverineav has joined #openstack-infra | 11:00 | |
| *** portdirect has joined #openstack-infra | 11:01 | |
| *** spsurya has joined #openstack-infra | 11:01 | |
| *** whoami-rajat has joined #openstack-infra | 11:01 | |
| *** dougwig has joined #openstack-infra | 11:01 | |
| *** wolverineav has quit IRC | 11:05 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: Elasticsearch Zuul reporter https://review.openstack.org/644927 | 11:12 |
| *** kaisers has quit IRC | 11:12 | |
| *** kaisers has joined #openstack-infra | 11:13 | |
| *** notmyname has quit IRC | 11:28 | |
| *** notmyname has joined #openstack-infra | 11:30 | |
| *** arxcruz|pto is now known as arxcruz | 11:31 | |
| *** Lucas_Gray has quit IRC | 11:34 | |
| *** rpioso|afk is now known as rpioso | 11:46 | |
| *** pcaruana has quit IRC | 11:53 | |
| openstackgerrit | Merged openstack-infra/devstack-gate master: Replace openstack.org git:// URLs with https:// https://review.openstack.org/645451 | 12:00 |
| *** dtantsur|brb is now known as dtantsur | 12:05 | |
| *** jento_ has joined #openstack-infra | 12:06 | |
| *** JpMaxMan_ has joined #openstack-infra | 12:07 | |
| *** davidlenwell_ has joined #openstack-infra | 12:07 | |
| *** csatari_ has joined #openstack-infra | 12:07 | |
| *** Guest12731 has joined #openstack-infra | 12:08 | |
| *** roman_g has quit IRC | 12:09 | |
| *** kgiusti has joined #openstack-infra | 12:11 | |
| *** rh-jelabarre has joined #openstack-infra | 12:12 | |
| *** kopecmartin|lunc is now known as kopecmartin | 12:14 | |
| *** rlandy has joined #openstack-infra | 12:14 | |
| *** eharney has quit IRC | 12:14 | |
| *** logan- has quit IRC | 12:14 | |
| *** ab-a has quit IRC | 12:14 | |
| *** davidlenwell has quit IRC | 12:14 | |
| *** JpMaxMan has quit IRC | 12:14 | |
| *** csatari has quit IRC | 12:14 | |
| *** jento has quit IRC | 12:14 | |
| *** Guest12731 is now known as logan- | 12:14 | |
| *** csatari_ is now known as csatari | 12:14 | |
| *** jento_ is now known as jento | 12:14 | |
| *** davidlenwell_ is now known as davidlenwell | 12:14 | |
| *** JpMaxMan_ is now known as JpMaxMan | 12:14 | |
| *** weshay is now known as weshay|rover | 12:17 | |
| openstackgerrit | Luigi Toscano proposed openstack-infra/zuul-jobs master: DNM Debug stage-output, change archival mechanism https://review.openstack.org/645239 | 12:19 |
| *** eharney has joined #openstack-infra | 12:23 | |
| *** trown|outtypewww is now known as trown|brb | 12:24 | |
| *** trown|brb is now known as trown | 12:24 | |
| *** jbadiapa has quit IRC | 12:33 | |
| *** jpena is now known as jpena|lunch | 12:36 | |
| *** udesale has quit IRC | 12:47 | |
| *** udesale has joined #openstack-infra | 12:48 | |
| fungi | ianw: regarding http://paste.openstack.org/show/748220/ i think it's fair to say that no bug tracker (that i'm aware of at least) has a good way to track hundreds of distinct tasks in one report. what we discovered is that storyboard handles it better than, say, launchpad. it does render after a bit of a wait rather than just throwing up an api timeout error and asking you to try again later | 12:49 |
| Shrews | fungi: the nodepool.yaml change we merged yesterday for the builders still has not propogated to the nb01/nb02. any reason you may know of? | 12:51 |
| fungi | nothing comes to mind but i'll take a look and see if i can figure out why | 12:54 |
| *** pcaruana has joined #openstack-infra | 12:54 | |
| fungi | last puppet apply was 16:58:38z | 12:56 |
| fungi | (on nb01) | 12:56 |
| fungi | ansible is still connecting to it regularly though | 12:58 |
| fungi | so i have a feeling we broke something with host matching around that time | 12:58 |
| Shrews | hrm | 12:58 |
| fungi | digging into recent config changes now to see what jumps out at me | 12:58 |
| fungi | 643713 was the last change that messed with host globs but it merged at 15:41z so the delay for it to break matching doesn't quite fit | 13:00 |
| *** altlogbot_0 has quit IRC | 13:01 | |
| *** irclogbot_1 has quit IRC | 13:01 | |
| *** irclogbot_2 has joined #openstack-infra | 13:02 | |
| *** altlogbot_3 has joined #openstack-infra | 13:02 | |
| *** lseki has joined #openstack-infra | 13:02 | |
| *** Lucas_Gray has joined #openstack-infra | 13:03 | |
| fungi | according to run_all_cron.log on bridge.o.o, the "puppet : copy puppet modules" task is failing | 13:04 |
| fungi | on nb01 and nb02 | 13:04 |
| fungi | could it be that we copy modules into /opt which is full? | 13:06 |
| *** dklyle has quit IRC | 13:07 | |
| fungi | strangely it's not breaking on nb03 even though /opt is full there too | 13:07 |
| Shrews | hrm, that could be. nb03 has a separate config | 13:08 |
| openstackgerrit | Luigi Toscano proposed openstack-infra/zuul-jobs master: DNM Debug stage-output, change archival mechanism https://review.openstack.org/645239 | 13:08 |
| fungi | oh, i see | 13:08 |
| fungi | the /opt fs is only "full" on nb03 but still has some space writeable by root | 13:08 |
| fungi | the /opt fs on nb01 and nb02 really have no remaining available blocks even for root | 13:09 |
| Shrews | chicken+egg=uhoh | 13:09 |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: Elasticsearch Zuul reporter https://review.openstack.org/644927 | 13:09 |
| fungi | so if we temporarily free up some space in /opt on those two i think things will start working again | 13:09 |
| zigo | fungi: Hi there! Is there a Buster image in infra already? | 13:10 |
| Shrews | fungi: maybe we can delete one of the older .raw files for a current image?? | 13:12 |
| *** dklyle has joined #openstack-infra | 13:12 | |
| fungi | Shrews: that ought to be plenty | 13:12 |
| fungi | zigo: not to my knowledge | 13:12 |
| zigo | :/ | 13:13 |
| zigo | fungi: I'll need it for unit testing Stein on puppet-openstack ... | 13:13 |
| * zigo needs to start a patch then. | 13:13 | |
| fungi | we can try adding it shortly, but need to clean up some space on our nodepool image builders first (which is what Shrews is working on right now) | 13:13 |
| zigo | fungi: Ok, thanks. | 13:14 |
| zigo | Can I attempt a patch anyway? | 13:14 |
| fungi | of course! | 13:14 |
| zigo | fungi: It's probably going to be wrong, but I'll try! :P | 13:14 |
| zigo | (ie: search for stretch and try to replicate ...) | 13:14 |
| fungi | just warning that if it takes us a little longer to fix the current image retention issues we're dealing with we may hold off approving the patch even if it's correct | 13:15 |
| fungi | though we hope it'll self-correct here within a few hours | 13:15 |
| zigo | fungi: I've just started the Stein packaging, so it's not urgent, I wont start using before like 1 week or 2... | 13:15 |
| fungi | sounds exciting for sure | 13:15 |
| zigo | :) | 13:16 |
| fungi | thanks for working on that! | 13:16 |
| zigo | fungi: Have you heard of this? https://salsa.debian.org/openstack-team/debian/openstack-cluster-installer | 13:16 |
| zigo | That's my own tool, based on Debian & puppet-openstack. It's part of Buster ... :P | 13:16 |
| zigo | We're currently using it in production @my-work. | 13:17 |
| fungi | the name sounds familiar but i didn't realize that was what you were working on | 13:17 |
| zigo | I've done it all. | 13:17 |
| zigo | :) | 13:17 |
| zigo | (nearly completely alone) | 13:17 |
| zigo | That's the reason why I care about puppet-openstack gating for Debian Buster. | 13:18 |
| fungi | makes sense | 13:18 |
| fungi | so basically a push-button openstack installer which uses the puppet-openstack modules? | 13:18 |
| Shrews | fungi: perhaps we should kick off puppet now before the running dib process take the free space | 13:19 |
| zigo | fungi: Yeah. Though these days, I only use the cli, there's also a web interface for it. | 13:19 |
| Shrews | (which seems to be happening) | 13:19 |
| fungi | Shrews: i'll use kick.sh now on them | 13:19 |
| zigo | fungi: I try to make it simple and stupid, no OOO, containers, or such. | 13:19 |
| *** eharney has quit IRC | 13:20 | |
| zigo | Having it working is good enough, IMO ! :) | 13:20 |
| zigo | (and actually, not that easy to do...) | 13:20 |
| *** jbadiapa has joined #openstack-infra | 13:20 | |
| fungi | zigo: yeah, i meant "push-button" figuratively, not necessarily implying actual buttons | 13:20 |
| zigo | :) | 13:20 |
| zigo | fungi: Do I need to edit nodepool/n*.openstack.org.yaml in my patch? | 13:21 |
| zigo | And Grafana? | 13:21 |
| fungi | Shrews: okay, on bridge.o.o i did `sudo /opt/system-config/tools/kick.sh nb01*:nb02*` and it's updating them now | 13:23 |
| fungi | zigo: yes, the nodepool files are what will configure building and uploading the images and the grafana files are how we'll graph usage and availability for those new node labels | 13:24 |
| fungi | Shrews: the "puppet : run puppet" task just completed on nb01 | 13:25 |
| fungi | nb02 seems to be taking longer | 13:26 |
| fungi | and finally completed on nb02 now as well | 13:26 |
| fungi | Shrews: should hopefully be starting to delete the old images now? | 13:26 |
| openstackgerrit | Luigi Toscano proposed openstack-infra/zuul-jobs master: DNM Debug stage-output, change archival mechanism https://review.openstack.org/645239 | 13:26 |
| Shrews | fungi: yep. disk is freeing up too | 13:27 |
| fungi | zigo: the grafana config addition is what will make it show up on graphs like the ones you see at http://grafana.openstack.org/dashboard/db/nodepool | 13:28 |
| Shrews | fungi: /opt on nb01 now around 62% in use, and 77% on nb02 | 13:29 |
| fungi | great! | 13:31 |
| Shrews | cleanup seems to be done, stabilized on those #s | 13:32 |
| Shrews | yay auto-fixing of things | 13:32 |
| fungi | except when there's a catch-22 where the breakage prevents applying the fix | 13:33 |
| *** jpena|lunch is now known as jpena | 13:36 | |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config master: Add a Debian Buster image. https://review.openstack.org/645574 | 13:36 |
| zigo | fungi: Is there some other place I should commit as well, or is this repo enough? | 13:36 |
| fungi | looking | 13:36 |
| zigo | Like, system-config also maybe? | 13:37 |
| fungi | zigo: yeah, modules/openstack_project/manifests/mirror_update.pp is where we'd add the reprepro configuration for maintaining a debian mirror cache in each nodepool provider | 13:42 |
| fungi | should be able to add a gnupg_key resource for the appropriate signing key and add change the releases lists in appropriate reprepro resources to something like ['stretch', 'buster'] | 13:45 |
| fungi | the other existing occurrences of "stretch" in openstack-infra/system-config can be safely ignored for now i think, as those are more related to container images we're building for some of our newer opendev services | 13:47 |
| fungi | (opendev.org for example is actually running in debian/stretch-based docker containers) | 13:48 |
| Shrews | fungi: seeing how each new image (e.g. ^) adds at least 3 concurrent on-disk images to /opt, we might want to begin considering a path to increasing disk space in /opt in the not-too-far-away future | 13:49 |
| fungi | i concur | 13:49 |
| *** mriedem has joined #openstack-infra | 13:51 | |
| fungi | Shrews: looks like /opt is already in lvm2 logvols on cinder volumes, so should just be a matter of attaching more cinder volumes and growing the vg/lv/fs | 13:51 |
| *** Lucas_Gray has quit IRC | 13:51 | |
| Shrews | cool | 13:52 |
| *** Lucas_Gray has joined #openstack-infra | 13:52 | |
| fungi | Shrews: oh, except on nb03 | 13:53 |
| fungi | where we may need to consider other options (not sure that arm64 linaro cloud has cinder available) | 13:53 |
| *** Lucas_Gray has quit IRC | 13:56 | |
| *** efried is now known as fried_rice | 13:58 | |
| *** Lucas_Gray has joined #openstack-infra | 13:59 | |
| *** bnemec is now known as beekneemech | 13:59 | |
| *** jaosorior has quit IRC | 14:01 | |
| *** jbadiapa has quit IRC | 14:04 | |
| *** eharney has joined #openstack-infra | 14:06 | |
| *** iurygregory has quit IRC | 14:11 | |
| *** iurygregory has joined #openstack-infra | 14:13 | |
| *** iurygregory has quit IRC | 14:24 | |
| *** iurygregory has joined #openstack-infra | 14:24 | |
| *** armax has joined #openstack-infra | 14:29 | |
| clarkb | Shrews: fungi when I've looked at this in the past the issue is we've leaked images to disk | 14:35 |
| clarkb | have we checked all images on disk arevalid according to nodepool? | 14:35 |
| fungi | i have not. but also i suspect we're capped at ~200gb /opt on nb03 without rebuilding on a different flavor | 14:36 |
| clarkb | the arm builder shouldnt need much disk | 14:36 |
| clarkb | it builds ~3 qcow2 only images | 14:37 |
| fungi | ahh, then mayhaps it has a lot of leaked images | 14:37 |
| fungi | because it's also basically full | 14:37 |
| clarkb | one theory that cameup before was whe we restart the builder the current image build may leak | 14:39 |
| *** cmoura has quit IRC | 14:42 | |
| *** cmoura has joined #openstack-infra | 14:43 | |
| *** lmiccini has joined #openstack-infra | 15:01 | |
| *** armax has quit IRC | 15:03 | |
| lmiccini | o/ I am facing something similar to https://bugs.launchpad.net/openstack-ci/+bug/1394191 with my gerrit account, anyone able to help me out? | 15:07 |
| openstack | Launchpad bug 1394191 in OpenStack Core Infrastructure "can't be added as a gerrit reviewer " [Medium,Fix released] - Assigned to Jeremy Stanley (fungi) | 15:07 |
| *** harlowja has joined #openstack-infra | 15:07 | |
| fungi | lmiccini: sure, i can take a look, it's almost always the result of having multiple gerrit accounts with the same e-mail address | 15:08 |
| lmiccini | fungi: thanks! I think I've tried too hard to work around it myself and ended up with a bunch of duplicates | 15:09 |
| *** jamesmcarthur has joined #openstack-infra | 15:12 | |
| fungi | lmiccini: indeed, i find 4 different accounts in gerrit for someone with the same username as your irc nick | 15:12 |
| fungi | account numbers 19705, 25259, 25412 and 30149 | 15:13 |
| *** dpawlik has quit IRC | 15:14 | |
| lmiccini | fungi: I have a "lmiccini2" with ID 30126 that I've tried to merge duplicate accounts into (and apparently succeeded). any chance you can wipe those out? | 15:14 |
| fungi | yikes, there are also 2 accounts with the same e-mail address for username:lmiccini2 | 15:16 |
| fungi | 23817 and 30126 | 15:16 |
| lmiccini | fungi: ouch | 15:16 |
| *** cgoncalves has quit IRC | 15:17 | |
| lmiccini | fungi: wipe everything out maybe? I don't care about history or anything, just want to clean up things | 15:17 |
| fungi | are you a member of any core review groups in gerrit, or do you have any open changes in review? membership/ownership of those will be lost if i deactivate the accounts used for them | 15:17 |
| lmiccini | fungi: nope all closed | 15:17 |
| fungi | okay, i'll deactivate the following accounts: 19705, 23817, 25259, 25412 and 30149 | 15:18 |
| fungi | that will leave 30126 as your only active account | 15:18 |
| *** cgoncalves has joined #openstack-infra | 15:18 | |
| lmiccini | fungi: awesome thanks | 15:18 |
| fungi | cool, doing that now | 15:19 |
| fungi | lmiccini: all done, let us know if you run into any issues with this | 15:19 |
| lmiccini | fungi: will do. thank you very much | 15:19 |
| fungi | you may want to log out of gerrit and ubuntuone and log back into them again just to make sure, and double-check that you can push a change | 15:20 |
| fungi | #status log deactivated duplicate gerrit accounts 19705, 23817, 25259, 25412 and 30149 at the request of lmiccini | 15:21 |
| openstackstatus | fungi: finished logging | 15:21 |
| *** altlogbot_3 has quit IRC | 15:21 | |
| *** aaronsheffield has joined #openstack-infra | 15:21 | |
| clarkb | Shrews: fungi http://paste.openstack.org/show/748248/ these images are all leaked on nb03 | 15:22 |
| clarkb | I don't think we should worry about adding disk or replacing builders where we can't add disk until we understand the image leak problem | 15:22 |
| fungi | clarkb: makes sense, thanks for checking | 15:22 |
| fungi | also noting that /opt/dib_tmp on nb03 has 61G of data in it right now... mostly stale? | 15:23 |
| clarkb | fungi: likely | 15:23 |
| fungi | tons of profiledir.* directories | 15:24 |
| clarkb | I haven't cleaned up those leaked miages in case shrews wants to look closer but I can help clean them up in a bit | 15:24 |
| clarkb | fungi: if you have a sec care to review https://review.openstack.org/#/c/645372/1 and children for more puppet 4 good ness? | 15:24 |
| * clarkb finds breakfast | 15:25 | |
| *** altlogbot_2 has joined #openstack-infra | 15:25 | |
| aaronsheffield | Has zuul changed recently (this week) that would have removed one or more of the following variables? zuul.branch, zuul.change, zuul.newrev, zuul.patchset? Airship gates are failing on code like https://github.com/openstack/airship-shipyard/blob/c7c25e8cdafa34a04419a2740e7636631f37404b/tools/gate/roles/build-images/tasks/airship-shipyard.yaml#L28, an example is https://review.openstack.org/#/c/644958/ | 15:26 |
| lmiccini | fungi: tested and working fine. thanks again | 15:27 |
| fungi | aaronsheffield: the default ansible version used in zuul jobs has increased from 2.5 to 2.7 | 15:29 |
| fungi | aaronsheffield: and ansible 2.7 is a little more picky about not ignoring missing variables | 15:29 |
| *** irclogbot_2 has quit IRC | 15:30 | |
| aaronsheffield | Gotcha, so we probably had a missing variable for a long time, but just now a problem. | 15:30 |
| fungi | in particular this is not the first incident we've seen with jobs failing on "The field 'environment' has an invalid value, which includes an undefined variable. The error was: 'dict object' has no attribute 'newrev' | 15:30 |
| fungi | in the check/gate pipelines | 15:30 |
| clarkb | zuul only sets those values when appropriate for the event that triggered the job | 15:30 |
| clarkb | so post pipeline jobs don't get a branch (neither do release jobs) | 15:31 |
| openstackgerrit | Luigi Toscano proposed openstack-infra/zuul-jobs master: stage-output: fix the archiving of all files https://review.openstack.org/645239 | 15:31 |
| fungi | and check/gate pipeline jobs don't have a zuul.newrev | 15:31 |
| tosky | fungi: ^^ that review should fix the log compression phase of stage-output | 15:32 |
| fungi | aaronsheffield: but yes, that missing variable was simply being ignored by ansible 2.5 (or only induced it to emit a non-failure warning) | 15:32 |
| *** irclogbot_3 has joined #openstack-infra | 15:32 | |
| *** michaelbeaver has joined #openstack-infra | 15:32 | |
| aaronsheffield | thanks for the quick response. | 15:32 |
| fungi | aaronsheffield: you *can* temporarily downgrade the version of ansible in use for that job if needed, but be warned that when ansible 2.5 reaches end of life in a few weeks we expect to drop it from the available versions on our executors | 15:33 |
| fungi | aaronsheffield: also see http://lists.openstack.org/pipermail/openstack-discuss/2019-March/004034.html | 15:36 |
| *** irclogbot_3 has quit IRC | 15:36 | |
| fungi | i thought we'd also posted that to the openstack-infra ml but i guess we didn't | 15:36 |
| fungi | anyway, running late for a lunch appointment. should be back shortly | 15:37 |
| *** irclogbot_2 has joined #openstack-infra | 15:37 | |
| fungi | infra-root: citycloud has sent us another notice about the impending la1 region shutdown, stating that we've booted instances there since the previous notice ~ a month ago. that seems... unlikely to me since it's had max-servers 0 since the beginning of september. maybe they're counting image uploads? it's probably time to remove them from our nodepool configuration cleanly | 15:40 |
| fungi | and with that, i disappear for lunch | 15:40 |
| *** pgaxatte has quit IRC | 15:52 | |
| openstackgerrit | Logan V proposed openstack-infra/project-config master: Revert "Disable provider limestone" https://review.openstack.org/645639 | 15:53 |
| openstackgerrit | Merged openstack-infra/system-config master: Fix groups.openstack.org glob https://review.openstack.org/645326 | 15:58 |
| Shrews | fungi: clarkb: i can examine the nb03 logs to see if i can spot the leak issue. will do that in a bit | 16:07 |
| *** mriedem is now known as mriedem_afk | 16:07 | |
| *** dustinc has joined #openstack-infra | 16:08 | |
| clarkb | Shrews: ok should we hold off on deleting leaked images then? or proceed with taht (but record them first) so that builds will work again? | 16:11 |
| Shrews | clarkb: go ahead, don't need those, just the logs | 16:11 |
| clarkb | ok I'm starting with the list I posted for nb03 and will generate them for nb01 and nb02 as well | 16:12 |
| *** Lucas_Gray has quit IRC | 16:13 | |
| clarkb | nb03 now at /dev/sdb 197G 101G 87G 54% /opt | 16:15 |
| *** helenaAM has quit IRC | 16:15 | |
| *** ykarel_ has joined #openstack-infra | 16:18 | |
| clarkb | http://paste.openstack.org/show/748254/ is nb02's list | 16:19 |
| *** imacdonn has quit IRC | 16:19 | |
| *** imacdonn has joined #openstack-infra | 16:20 | |
| *** ykarel has quit IRC | 16:20 | |
| *** janki has quit IRC | 16:22 | |
| clarkb | nb02 now at /dev/mapper/main-nodepool 1008G 295G 714G 30% /opt | 16:23 |
| *** yamamoto has quit IRC | 16:24 | |
| *** ykarel_ is now known as ykarel | 16:28 | |
| clarkb | http://paste.openstack.org/show/748255/ nb01's leaked file list | 16:28 |
| clarkb | nb01 now at /dev/mapper/main-nodepool 1008G 635G 374G 63% /opt | 16:29 |
| *** yamamoto has joined #openstack-infra | 16:30 | |
| *** ykarel is now known as ykarel|away | 16:30 | |
| *** fried_rice is now known as fried_rolls | 16:30 | |
| clarkb | infra-root can I get reviews on https://review.openstack.org/#/c/645372/1 and children? I'd like to get that stack in today before I'm afk most of next week | 16:31 |
| clarkb | gets a large chunk of our servers onto puppet4 | 16:32 |
| *** roman_g has joined #openstack-infra | 16:33 | |
| *** yamamoto has quit IRC | 16:35 | |
| *** sthussey has joined #openstack-infra | 16:38 | |
| *** rpittau is now known as rpittau|afk | 16:39 | |
| clarkb | corvus: your mailman queue cleanup command failed, but I believe that is because there was nothing to cleanup. Cna you check http://paste.openstack.org/show/748259/ and see if that is what you see as well? | 16:39 |
| *** dpawlik has joined #openstack-infra | 16:42 | |
| *** iurygregory has quit IRC | 16:45 | |
| *** dpawlik has quit IRC | 16:47 | |
| *** e0ne has joined #openstack-infra | 16:48 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Update even more servers to puppet4 https://review.openstack.org/645375 | 16:51 |
| *** jpich has quit IRC | 16:54 | |
| Shrews | clarkb: oh weird. so it seems those leaked files are being created (or at least last written to) *after* the file cleanup phase runs | 16:54 |
| Shrews | based on timestamps | 16:55 |
| clarkb | Shrews: huh I wonder if dib is syncing those files to disk after we think it is done somehow | 16:55 |
| Shrews | so i guess the creating process still has them open when we try to read them | 16:55 |
| clarkb | ya | 16:55 |
| Shrews | ya | 16:55 |
| clarkb | Shrews: maybe we can check that dib has completely exited before cleaning up? | 16:56 |
| Shrews | maybe | 16:56 |
| clarkb | (could have sigchild queue up the cleanup? | 16:56 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Update component diagram to show statsd https://review.openstack.org/645798 | 16:58 |
| *** e0ne has quit IRC | 16:59 | |
| Shrews | this seems to be related to losing the zookeeper connection | 16:59 |
| *** udesale has quit IRC | 17:00 | |
| Shrews | which maybe causes us to lose our image build lock in zk... which is how we test to see if the build is inprogress | 17:01 |
| corvus | clarkb: yes, it's normal for those directories to be empty | 17:01 |
| *** chandankumar is now known as raukadah | 17:02 | |
| clarkb | corvus: it is safe for mailman to start in that state? I expect the systemd switch post upgrade to cause that to happen | 17:03 |
| corvus | clarkb: yes | 17:04 |
| corvus | clarkb: just means there's nothing for it to do | 17:04 |
| clarkb | perfect, thanks | 17:04 |
| *** jamesmcarthur has quit IRC | 17:05 | |
| corvus | fungi: does my comment on https://review.openstack.org/645346 answer your question? | 17:06 |
| fungi | i shall find out now! | 17:07 |
| fungi | (as my lunch is presently digesting) | 17:09 |
| corvus | fungi: also replied on https://review.openstack.org/645391 | 17:09 |
| fungi | thanks! | 17:09 |
| *** dtantsur is now known as dtantsur|afk | 17:11 | |
| *** trown is now known as trown|lunch | 17:11 | |
| *** gfidente has quit IRC | 17:14 | |
| *** diablo_rojo has joined #openstack-infra | 17:15 | |
| *** ramishra_ has quit IRC | 17:17 | |
| *** jamesmcarthur has joined #openstack-infra | 17:22 | |
| *** michaelbeaver has quit IRC | 17:23 | |
| *** michael-beaver has joined #openstack-infra | 17:23 | |
| *** mattw4 has joined #openstack-infra | 17:25 | |
| *** eharney has quit IRC | 17:26 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add fetch-sphinx-tarball role https://review.openstack.org/645346 | 17:26 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add download artifact role https://review.openstack.org/645384 | 17:26 |
| *** derekh has quit IRC | 17:26 | |
| *** jamesmcarthur has quit IRC | 17:27 | |
| *** tjgresha has joined #openstack-infra | 17:29 | |
| fungi | c | 17:30 |
| fungi | wait, this is not my mutt window | 17:30 |
| *** roman_g has quit IRC | 17:30 | |
| *** tjgresha has quit IRC | 17:30 | |
| pabelanger | I happen to notice a zuul config error: https://zuul.openstack.org/config-errors FYI | 17:31 |
| fungi | http://git.openstack.org/cgit/openstack/masakari/tree/.zuul.yaml | 17:33 |
| *** tjgresha_ has joined #openstack-infra | 17:34 | |
| *** jamesmcarthur has joined #openstack-infra | 17:34 | |
| pabelanger | looks like the nodeset is defined more then once | 17:35 |
| pabelanger | and when bionic changes landed, zuul raised error because they are now different | 17:35 |
| *** tjgresha has joined #openstack-infra | 17:37 | |
| *** tjgresha_ has left #openstack-infra | 17:37 | |
| *** tjgresha_ has quit IRC | 17:38 | |
| *** tjgresha has quit IRC | 17:38 | |
| *** tjgresha has joined #openstack-infra | 17:39 | |
| *** diablo_rojo has quit IRC | 17:39 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/bindep master: Expose base python version as an atom https://review.openstack.org/639951 | 17:44 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix ignored default ansible version https://review.openstack.org/645819 | 17:47 |
| *** psachin has quit IRC | 17:48 | |
| *** ykarel_ has joined #openstack-infra | 17:50 | |
| *** gmann is now known as gmann_afk | 17:52 | |
| *** ykarel|away has quit IRC | 17:52 | |
| *** mriedem_afk is now known as mriedem | 17:55 | |
| *** tjgresha has quit IRC | 17:56 | |
| *** tjgresha has joined #openstack-infra | 17:56 | |
| *** trown|lunch is now known as trown | 18:02 | |
| clarkb | infra-root I've upgraded my snapshotted lists.o.o server per https://etherpad.openstack.org/p/lists.o.o-trusty-to-xenial 166.78.27.52 is its ip address if you would like to log in and look around | 18:08 |
| clarkb | I've recordred my notes on the etherpad. I think the first thing I am goign to look into is why systemct reports errors connecting to the upstart socket | 18:09 |
| *** gmann_afk is now known as gmann | 18:11 | |
| *** jpena is now known as jpena|off | 18:13 | |
| slaweq | clarkb: hi | 18:14 |
| clarkb | /sbin/init is a link to systemd so we have properly converted over to systemd | 18:14 |
| clarkb | slaweq: hello | 18:14 |
| slaweq | clarkb: I saw today at list couple of times errors like http://logs.openstack.org/86/643486/12/check/neutron-tempest-plugin-scenario-linuxbridge/48055f7/controller/logs/devstacklog.txt.gz#_2019-03-22_17_00_19_424 during devstack | 18:14 |
| slaweq | is it something You are aware of? | 18:14 |
| clarkb | slaweq: that is news to me, but the jobs should use our mirrors which are curated (and should never have those errors, its why we run our own mirrors and only "publish" them when the mirror is viable) | 18:15 |
| *** tjgresha has quit IRC | 18:15 | |
| clarkb | slaweq: http://logs.openstack.org/86/643486/12/check/neutron-tempest-plugin-scenario-linuxbridge/48055f7/controller/logs/devstacklog.txt.gz#_2019-03-22_16_40_19_946 shows the job starts out using our mirrors | 18:16 |
| clarkb | slaweq: is somethign in the job overriding the apt config? | 18:16 |
| *** tjgresha has joined #openstack-infra | 18:16 | |
| *** xek_ has quit IRC | 18:17 | |
| *** ykarel_ has quit IRC | 18:18 | |
| clarkb | slaweq: that is failing in the customization of the nested ubuntu test image | 18:19 |
| *** jamesmcarthur has quit IRC | 18:19 | |
| clarkb | slaweq: so it is outside of infras control unless you configure it to use our mirrors as well | 18:19 |
| *** armax has joined #openstack-infra | 18:20 | |
| *** jamesmcarthur has joined #openstack-infra | 18:20 | |
| *** ginopc has quit IRC | 18:20 | |
| clarkb | infra-root my naive read of the mailman upgrade is that our vhost configs have survived the upgrade process | 18:21 |
| slaweq | clarkb: ahh, ok | 18:22 |
| slaweq | now I remember that we did such tool to customize image before upload to devstack's glance | 18:23 |
| slaweq | so it's on our side bug then | 18:23 |
| fungi | clarkb: it seems to be working for me after an /etc/hosts override | 18:23 |
| slaweq | thanks a lot for help | 18:23 |
| *** jamesmcarthur has quit IRC | 18:24 | |
| clarkb | slaweq: unfortunately deb repos suffer from an updating flaw where you can have package and index mismatches leading to unhappy apt-get installs | 18:24 |
| fungi | clarkb: last message at http://lists.openstack.org/pipermail/openstack-discuss/2019-March/date.html#start was sent Thu Mar 21 17:48:38 UTC 2019 | 18:24 |
| fungi | (in the snapshot) | 18:24 |
| clarkb | slaweq: this is why we build our own mirrors on top of afs and only publish updates once we have done verification taht the mirror is valid | 18:24 |
| clarkb | fungi: cool I don't have exim or mailman running on the server yet, but good to know the http side of things is happy | 18:25 |
| corvus | fungi, clarkb: the error in https://review.openstack.org/645391 is "neat" (post review pipeline gets dynamic config for trusted projects, but trusted project git repos aren't updated until changes land) | 18:25 |
| slaweq | clarkb: thx for explanation | 18:25 |
| fungi | clarkb: also http://lists.zuul-ci.org/ seems to be working and serving archives on it too | 18:25 |
| slaweq | clarkb: I will have to check why we are using this customized image in this job and (maybe) switch it to use OS mirrors inside it then | 18:25 |
| slaweq | clarkb: thx a lot for help | 18:25 |
| clarkb | slaweq: no problem | 18:25 |
| fungi | corvus: yeah, when i saw the gerrit notification of your second workflow +1 i got curious and went looking at the error. made sense to me | 18:26 |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: Add opendev docs build/promote jobs https://review.openstack.org/645391 | 18:27 |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: Use new opendev docs jobs https://review.openstack.org/645832 | 18:27 |
| corvus | fungi, clarkb: ^ those 2 changes should be very easy to review :) | 18:28 |
| corvus | i just moved the project stanza update into its own change | 18:28 |
| clarkb | systemd-sysv depends on upstart on xenial. I'm guessing that is to compat layer in old upstart jobs? So the fix for that isn't to just remove the upstart package :/ | 18:29 |
| fungi | clarkb: yeah, debian derivatives can have as many init systems installed as you want | 18:30 |
| fungi | what matters is which one gets invoked by the kernel at boot | 18:30 |
| *** armax has quit IRC | 18:31 | |
| fungi | it's not that uncommon (and was quite common for a while during the great systemd scourge) to have different boot stanzas which booted the same kernel and rootfs with various different inits | 18:31 |
| clarkb | fungi: ya https://unix.stackexchange.com/questions/429032/initctl-unable-to-connect-to-upstart says the issue is /etc/init.d/screen-cleanup being a symlink to upstart-job | 18:31 |
| clarkb | sure enough we have such a link | 18:32 |
| clarkb | we don't have it on the afs servers where things were happy | 18:32 |
| fungi | sounds like the case, agreed | 18:32 |
| *** armax has joined #openstack-infra | 18:33 | |
| clarkb | the screen package apparently installs that file | 18:34 |
| clarkb | I can remove the file then reinstall screen to see if it doesn't set it up that way anymore? | 18:34 |
| clarkb | *remove the link | 18:34 |
| clarkb | oh ya there is a screen-cleanup.dpkg-new | 18:35 |
| clarkb | ok etherpad ammended on how to fix that item | 18:38 |
| *** eharney has joined #openstack-infra | 18:40 | |
| clarkb | I've double checked that mailman and exim aren't provided via systemd units now so our existing sysv scripts should continue to work as is for the vhosting in mailman | 18:42 |
| clarkb | and the exim config updates seem to be chagnes to macro listings in /etc/exim4/conf.d which I'm assuming we already work with on xenial in general (beacuse most of our machines are xenial with exim4) | 18:43 |
| clarkb | all this to say I've get to see any system level chagnes that should break our existing setup | 18:43 |
| clarkb | Should I reenable exim4 and mailman services then reboot to have them start and see if they are actually happy? | 18:44 |
| corvus | clarkb: "exim -bt openstack-discuss@lists.openstack.org" will do a very simple exim config check | 18:44 |
| clarkb | (as far as double checking an upgrade goes for exim and mailman I am sort of out of my element so input/ideas much appreciated) | 18:44 |
| clarkb | woot thanks | 18:44 |
| *** pcaruana has quit IRC | 18:45 | |
| clarkb | http://paste.openstack.org/show/748271/ that also appears to be happy | 18:46 |
| *** tjgresha has quit IRC | 18:50 | |
| *** tjgresha has joined #openstack-infra | 18:50 | |
| *** tjgresha_nope has joined #openstack-infra | 18:51 | |
| clarkb | there is apparently a way to set a site wide dmarc_moderation_action and if you do that then lists cannot set a less strict value | 18:52 |
| *** tjgresha_nope has quit IRC | 18:52 | |
| *** eharney has quit IRC | 18:52 | |
| *** tjgresha has quit IRC | 18:52 | |
| *** tjgresha has joined #openstack-infra | 18:53 | |
| *** eharney has joined #openstack-infra | 18:54 | |
| *** Adri2000 has quit IRC | 18:55 | |
| clarkb | ok what confuses me about ^ is that action always taken ? if so our strategy of accepting the email and not modifying it means that lists could still choose to munge or reject themselves :/ | 18:56 |
| clarkb | ya my reading of it is if we set it to Accept as default then a list will be able to set it to munge, wrap, reject, or discard | 18:58 |
| *** tjgresha_nope has joined #openstack-infra | 18:58 | |
| *** tjgresha has quit IRC | 18:58 | |
| openstackgerrit | Merged opendev/base-jobs master: Add opendev docs build/promote jobs https://review.openstack.org/645391 | 18:58 |
| *** tjgresha_nope has quit IRC | 18:58 | |
| *** tjgresha has joined #openstack-infra | 18:58 | |
| *** tjgresha has quit IRC | 18:59 | |
| *** tjgresha has joined #openstack-infra | 19:00 | |
| clarkb | I wonder if we can patch the html to prevent the option from being presented to people? | 19:00 |
| clarkb | They'd still be able to POST around it but maybe that is good enough? | 19:00 |
| fungi | or take it out of the message pipeline like we did with recipient deduplication | 19:01 |
| clarkb | https://wiki.list.org/DEV/DMARC is the docs fwiw and https://fossies.org/linux/mailman/Mailman/Defaults.py.in seems to be the listing of the defaults we can set | 19:03 |
| clarkb | This seems like a thnig that isn't going to get solved in a 10 minute brainstorm /me lets it stew in the back of the mind for a bit | 19:05 |
| clarkb | fungi: corvus https://review.openstack.org/#/c/645372/1 can I get a review on that please? | 19:05 |
| *** fried_rolls is now known as fried_rice | 19:07 | |
| clarkb | Back of brain thinking out loud: If we set the default to accept (whcih I think it already is) and we continue to pass email through for the most part as is, do we expect people to ever notice that is somethign that they can change and that they might want to cahnge it? Like maybe its enough to trust our users? though I guess as the user pool grows we won't necessarily keep up | 19:07 |
| fungi | it was enough for the listadmin of kata-dev to decide to set | 19:08 |
| clarkb | fungi: that was before we had a less bad answer to the problem though | 19:08 |
| *** tjgresha has quit IRC | 19:08 | |
| *** tjgresha has joined #openstack-infra | 19:09 | |
| *** tjgresha has quit IRC | 19:10 | |
| *** tjgresha_nope has joined #openstack-infra | 19:10 | |
| *** tjgresha_nope has quit IRC | 19:10 | |
| *** tjgresha has joined #openstack-infra | 19:10 | |
| clarkb | though I guess we still do have a few problem domains | 19:10 |
| clarkb | so people are likely to want to go investigating how to fix it | 19:10 |
| fungi | i'm still not entirely convinced our current solution is especially "less bad" though. it sacrifices recipient deduplication as well as a host of lesser mailman features, and still results in needing to disable subscription deactivation from bounces because 1. mailman mangles messages in other unconfigurable ways (whitespace normalization, references rewriting) which break dmarc, but worse 2. it | 19:11 |
| fungi | doesn't actually solve the case of people sending messages with broken dmarc signatures to the list either | 19:11 |
| fungi | s/dmarc/dkim/ really | 19:12 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Update query for bug 1820892 https://review.openstack.org/645851 | 19:13 |
| clarkb | have we confirmed any broken signatures as source of trouble? | 19:13 |
| openstack | bug 1820892 in devstack "Intermittent "Error starting thread.: ModuleNotFoundError: No module named 'etcd3gw'" in grenade-py3 jobs since March 14" [High,Fix released] https://launchpad.net/bugs/1820892 - Assigned to Matt Riedemann (mriedem) | 19:13 |
| corvus | i agree. i think the ideal would be to reject messages which mailman cannot safely process. | 19:13 |
| clarkb | (I wasn't sure if we ever pinned a failure down to that or not) | 19:13 |
| fungi | corvus had an idea of maybe also adding a receipt-time filter in exim which checks to see if the result of the transformations mailman performs will cause broken dkim sigs and then reject them before mailman gets them | 19:13 |
| corvus | (unfortunately mailman itself only has an option to reject all messages from dmarc domains) | 19:14 |
| fungi | i haven't spotted a dkim signature validation yet which i was able to attribute to being broken before it was handed off to mailman | 19:14 |
| fungi | er, dkim signature validation failure i mean | 19:14 |
| fungi | but that doesn't mean it can't happen | 19:14 |
| clarkb | corvus: ya that is my read of the dmarc_moderation_action config | 19:15 |
| clarkb | seems like mailman expects people to do munge or wrap | 19:15 |
| fungi | so even if we're able to coerce mailman into not altering anything which could possibly invalidate a dkim signature, we still have to take in to account that forwarding a message with an (accidentally or intentionally) invalid dkim signature could still disable subscriptions for a vast swath of subscribers | 19:16 |
| *** weshay|rover is now known as weshay | 19:17 | |
| corvus | it's very frustrating that the mailman folks have taken this approach -- since mailman knows, at the point that it sends out the message, whether it's going to work or not. | 19:17 |
| corvus | but that's not where the filtering happens | 19:17 |
| clarkb | that wasmy next question will mailman or exim not reject due to invlaid signature? | 19:18 |
| corvus | clarkb: exim could, but at that point the result is the same as the remote site rejecting -- a bounce to mailman | 19:18 |
| corvus | i mean, we could tell exim not to bounce | 19:19 |
| corvus | but then it's silently dropping | 19:19 |
| clarkb | can wehave exim bounce to the originator? | 19:19 |
| clarkb | I guess we have to have mailman update it forst | 19:19 |
| fungi | only feasibly if we can predict whether mailman will alter the message in ways that make the signature invalid | 19:19 |
| corvus | probably, though i'm not sure we could prevent multiple copies of that bounce, depending on how many copies of that message mailman emitted... | 19:20 |
| fungi | oh, after passing through mailman, yes perhaps, but again bounces aren't as useful as rejecting at rcpt command | 19:20 |
| corvus | yeah, that would be the real win | 19:21 |
| fungi | bounces are spoofable, whereas refusing receipt of the message at our server is less so | 19:21 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Add web / fingergw connections for components graph https://review.openstack.org/645852 | 19:21 |
| fungi | i wonder if the mailman pipeline scripts could be used in an exim filter | 19:22 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Increase default wait_timeout https://review.openstack.org/645853 | 19:22 |
| corvus | fungi: probably. we could probably come up with a python program which actually uses mailman internals to process a message. | 19:23 |
| fungi | like pass a copy o fthe message thruogh the pipeline parts which are likely to mangle the message and then through a dkim validator, and then accept or reject the message based on the results | 19:23 |
| corvus | if we can write that python program, we can *certainly* have exim run it and reject at rcpt time | 19:23 |
| *** jtomasek has quit IRC | 19:23 | |
| fungi | i wonder how pipeline-y the mm pipeline pieces are | 19:24 |
| clarkb | that doesnt affect a users ability to set munge or wrap but there wont be a reason too as all the emails that would get munged would be rejected upstream? | 19:25 |
| corvus | clarkb: right (assuming the lists are configured not to munge more than we're testing for) | 19:26 |
| fungi | the pipeline modules seem to be in /usr/lib/mailman/Mailman/Handlers/ | 19:26 |
| fungi | looks like they contain a process() function which takes the list, message and message data as parameters | 19:27 |
| fungi | and then they directly manipulate the message dict | 19:28 |
| corvus | i have to grab lunch; biab. | 19:28 |
| openstackgerrit | Merged opendev/base-jobs master: Use new opendev docs jobs https://review.openstack.org/645832 | 19:31 |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Update query for bug 1820892 https://review.openstack.org/645851 | 19:33 |
| openstack | bug 1820892 in devstack "Intermittent "Error starting thread.: ModuleNotFoundError: No module named 'etcd3gw'" in grenade-py3 jobs since March 14" [High,Fix released] https://launchpad.net/bugs/1820892 - Assigned to Matt Riedemann (mriedem) | 19:33 |
| *** mriedem has quit IRC | 19:46 | |
| *** mriedem has joined #openstack-infra | 19:47 | |
| *** yamamoto has joined #openstack-infra | 19:50 | |
| *** armax has quit IRC | 19:54 | |
| *** yamamoto has quit IRC | 19:56 | |
| *** armax has joined #openstack-infra | 19:59 | |
| *** armax has quit IRC | 20:02 | |
| *** armax has joined #openstack-infra | 20:06 | |
| *** kjackal has quit IRC | 20:15 | |
| openstackgerrit | Merged openstack-infra/system-config master: Run static and status under futureparser https://review.openstack.org/645372 | 20:15 |
| *** Lucas_Gray has joined #openstack-infra | 20:17 | |
| *** diablo_rojo has joined #openstack-infra | 20:22 | |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: Docs promotion: create destination directory https://review.openstack.org/645873 | 20:27 |
| corvus | fungi, clarkb: ^ that should fix an observed failure | 20:28 |
| *** armax has quit IRC | 20:31 | |
| openstackgerrit | Merged openstack/os-testr master: add python 3.7 unit test job https://review.openstack.org/637749 | 20:32 |
| *** Lucas_Gray has quit IRC | 20:33 | |
| openstackgerrit | Merged opendev/base-jobs master: Docs promotion: create destination directory https://review.openstack.org/645873 | 20:46 |
| *** armax has joined #openstack-infra | 20:50 | |
| *** trown is now known as trown|outtypewww | 20:58 | |
| clarkb | corvus: fungi: thinking about testing the lists server upgrade more, short of turning on exim and mailman is there anything else worth doing to test it? And if I turn on those processes is doing smtp over telnet going to be the easiest way to interact with it? | 21:01 |
| clarkb | (also I probabl won't get too much further into testing it today as I want to finish up the puppet4 thread before take monday-thursday off) | 21:01 |
| corvus | clarkb: i think that's it | 21:01 |
| clarkb | ok so all doable just may involve some rtfm'ing about smtp :) | 21:02 |
| clarkb | elo and rcpt is about all I currently remember and I probably got those wrong | 21:02 |
| corvus | clarkb: http://paste.openstack.org/show/748274/ | 21:04 |
| clarkb | thanks | 21:05 |
| corvus | i forgot to include the bits about getting an afs token in the promote job; i'll work on adding that now | 21:08 |
| corvus | i'll make a new principal for it | 21:10 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Don't assume secrets are text in encrypt_secret https://review.openstack.org/645888 | 21:22 |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: Obtain AFS creds in docs promote https://review.openstack.org/645893 | 21:32 |
| corvus | clarkb, fungi: ^ there we go | 21:32 |
| clarkb | corvus: zuul says there is an invalid byte | 21:32 |
| clarkb | I think zuul must expects the secrets to be utf8 internally? | 21:33 |
| clarkb | can we ask kerberos to use utf8 for its tokens? | 21:34 |
| corvus | clarkb: erm... hrm. i think this is how the others are done. | 21:34 |
| corvus | derp | 21:35 |
| corvus | we base64 encode the keytabs | 21:35 |
| corvus | maybe that "fix" to encrypt_secrets should be rethought as well | 21:35 |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: Obtain AFS creds in docs promote https://review.openstack.org/645893 | 21:37 |
| *** rkukura_ has joined #openstack-infra | 21:41 | |
| *** rkukura has quit IRC | 21:43 | |
| *** rkukura_ is now known as rkukura | 21:43 | |
| *** mgoddard has quit IRC | 21:47 | |
| *** mgoddard has joined #openstack-infra | 21:47 | |
| clarkb | infra-root static and status looked good under futureparser to switch to puppet 4 on them is on its way | 21:49 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Minor improvements to docker-image doc structure https://review.openstack.org/645897 | 21:49 |
| clarkb | https://review.openstack.org/#/c/645375/3 is next up and I can be around long enough today to ensure that that is happy once in | 21:49 |
| clarkb | that gives us puppet 4 on all the logstash hosts and mirror nodes | 21:50 |
| clarkb | we'll be down to a very small list of servers to update to puppet 4 once that is in :) | 21:50 |
| clarkb | fungi: corvus thank you! | 21:52 |
| *** e0ne has joined #openstack-infra | 21:54 | |
| *** auristor has quit IRC | 21:58 | |
| *** auristor has joined #openstack-infra | 21:58 | |
| openstackgerrit | Merged opendev/base-jobs master: Obtain AFS creds in docs promote https://review.openstack.org/645893 | 21:58 |
| openstackgerrit | Kendall Nelson proposed openstack-infra/storyboard-webclient master: Show Email Addresses when Searching https://review.openstack.org/589713 | 22:00 |
| clarkb | corvus: I'm trying to remember, last time we did a lists upgrade did we create a tests list or just use openstack-infra for that? | 22:02 |
| *** armax has quit IRC | 22:02 | |
| clarkb | I don't see a test list in the list of lists | 22:02 |
| clarkb | so I'm guessing we used the infra list for that | 22:02 |
| *** auristor has quit IRC | 22:03 | |
| corvus | clarkb: yeah i think we just used infra | 22:03 |
| corvus | we should really create a test list one of these days :) | 22:03 |
| clarkb | ++ | 22:04 |
| *** armax has joined #openstack-infra | 22:14 | |
| *** kgiusti has left #openstack-infra | 22:14 | |
| *** armax has quit IRC | 22:15 | |
| openstackgerrit | Merged openstack-infra/system-config master: Run static and status under puppet4 https://review.openstack.org/645373 | 22:17 |
| openstackgerrit | Merged openstack-infra/system-config master: Update even more servers to puppet4 https://review.openstack.org/645375 | 22:17 |
| *** michael-beaver has quit IRC | 22:20 | |
| *** auristor has joined #openstack-infra | 22:22 | |
| *** yamamoto has joined #openstack-infra | 22:27 | |
| clarkb | next run in half an hour should apply ^ those changes | 22:27 |
| corvus | clarkb, fungi: w00t: http://files.openstack.org/project/opendev.org/docs/opendev/base-jobs/latest/ | 22:31 |
| corvus | that's the opendev docs promotion working :) | 22:31 |
| *** yamamoto has quit IRC | 22:31 | |
| corvus | now we just need a vhost and dns and we're done | 22:31 |
| * fungi cheers | 22:31 | |
| clarkb | corvus: can probably update/replace the existing vhost on files? | 22:32 |
| corvus | yeah | 22:32 |
| corvus | i guess we're going to want an ssl cert | 22:32 |
| corvus | but we can probably skate by without one until ianw finished le | 22:32 |
| corvus | finishes le | 22:32 |
| *** e0ne has quit IRC | 22:36 | |
| corvus | that vhost was on files, right? | 22:36 |
| clarkb | corvus: yes it should be | 22:37 |
| clarkb | it was set up like zuul and starlingx afs hosted sites | 22:38 |
| corvus | ah! i found it :) | 22:39 |
| *** whoami-rajat has quit IRC | 22:41 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Serve docs.opendev.org from files.openstack.org https://review.openstack.org/645953 | 22:48 |
| corvus | clarkb: so that we don't have to immediately deal with the challenges of integrating letsencrypt with the pile of puppet that's files.o.o, we may want to go ahead and buy a comodo cert for that :/ | 22:48 |
| corvus | (i revised my opinon on that after making that change) | 22:49 |
| clarkb | corvus: ah does the assume https? | 22:50 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Serve docs.opendev.org from files.openstack.org https://review.openstack.org/645953 | 22:50 |
| clarkb | maybe that is something fungi can help with? (/me hoping to avoid needing to be around soon) | 22:50 |
| corvus | clarkb: yes; i could do a bunch of puppet to unassume that... i think it's okay to land that change and go aheand and serve it with the wrong cert for a little while | 22:50 |
| clarkb | k | 22:51 |
| corvus | mostly, i'm thinking that soon we really do want it to have the right cert, and because of the complexity of that host, we may not want to hang that on the letsencrypt work | 22:51 |
| clarkb | but ya if fungi can do the verification dance and expense report that would be good. I don't expect to have my laptop with me next week | 22:51 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Organize documentation by subject area https://review.openstack.org/645955 | 22:52 |
| *** tosky has quit IRC | 23:01 | |
| *** mriedem has quit IRC | 23:04 | |
| *** mattw4 has quit IRC | 23:05 | |
| *** rascasoft has quit IRC | 23:07 | |
| *** rlandy has quit IRC | 23:11 | |
| clarkb | cool there are some things that are a little unhappy with puppet4. logstash.o.o has the pip problem. Starting with that one | 23:11 |
| clarkb | hrm it is the pip problem but not due to the cryptography is slow warning. I'll need to trace this one | 23:13 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Explicitly set up mirror update crons under root user https://review.openstack.org/645959 | 23:27 |
| clarkb | that is the first fix | 23:27 |
| openstackgerrit | Kendall Nelson proposed openstack-infra/storyboard master: Link development.rst to contributing.rst https://review.openstack.org/645960 | 23:29 |
| clarkb | on status.o.o I had to run `npm rebuild node-sass` in /opt/openstack-health because the node/npm versions changed to the version we actually wanted | 23:30 |
| clarkb | I expect that will be happy on the next puppet run | 23:30 |
| clarkb | that leaves me with the weird pip behavior on logstash01.o.o trying to update gear via pip | 23:30 |
| clarkb | looking into that now | 23:31 |
| clarkb | on the crontab fix this is mostly cleaning up the puppet output as the default is to set it up for the user running puppet | 23:34 |
| clarkb | ok the logstash pip issue is our openstack_pip provider | 23:39 |
| clarkb | does anyone remember why we have that? | 23:39 |
| clarkb | confirmed status is happy after the npm/node sass fix | 23:42 |
| *** pcrews has quit IRC | 23:42 | |
| clarkb | on logstash01.o.o I've manaully upgraded gear to 0.13 I expect this will make puppet happe until we have to update gear. cmurphy mordred I think we should sort out whether or not openstack_pip is still useful before we update more puppet-4 hosts because hosts for things like nodepool and zuul depends on this quite a bit more than logstash | 23:44 |
| clarkb | I won't be around most of next week to sort that out, so sorry to not be a huge help with that | 23:44 |
| clarkb | cmurphy: mordred it is specifically erroring around https://git.openstack.org/cgit/openstack-infra/puppet-pip/tree/lib/puppet/provider/package/openstack_pip.rb#n15 | 23:49 |
| clarkb | http://paste.openstack.org/show/748277/ is what I was able to get out of debug tracing and I expect that if you downgrade gear to 0.12 you'll be able to reproduce | 23:50 |
| *** pcrews has joined #openstack-infra | 23:50 | |
| clarkb | cmurphy: mordred the regex at https://git.openstack.org/cgit/openstack-infra/puppet-pip/tree/lib/puppet/provider/package/openstack_pip.rb#n15 doesn't match new pip output. See http://paste.openstack.org/show/748278/ | 23:56 |
| clarkb | we can do `pip list --outdated --format columns` to get consistent output from both current and old (9.0.1 at least) pip | 23:57 |
| fungi | clarkb: corvus: sure, i'm happy to buy and expense a 1 year dv cert for docs.opendev.org next week or maybe over the weekend even | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!