| *** jamesmcarthur has joined #openstack-infra | 00:04 | |
| *** dhellmann has quit IRC | 00:05 | |
| *** dhellmann has joined #openstack-infra | 00:08 | |
| ianw | #define TYPE_DS 43 /* RFC 4033, 4034, and 4035 */ is somehow involved | 00:15 |
|---|---|---|
| clarkb | ianw: DS is a dns record type | 00:18 |
| clarkb | it validated child domain dnssec stuff iirc | 00:18 |
| ianw | clarkb: yeah, just from the backtrace i guess somehow dnssec is involved | 00:20 |
| *** jamesmcarthur has quit IRC | 00:20 | |
| ianw | i have a core file ... the only possible thing that *may* be in there i think would be the transfer key | 00:21 |
| clarkb | maybe opendev.org needs a DS record for acme.opendev.org ? | 00:21 |
| ianw | clarkb: i put in a change for that, which i think merged | 00:23 |
| ianw | we have some odd config on the nsd servers; it specifies zonefiles in dirs that don't exist ... i don't know if this matters | 00:23 |
| ianw | at any rate, nsd segfaulting is something i'm sure they'd like to fix, so i'm getting together a bug report | 00:24 |
| clarkb | ++ | 00:24 |
| clarkb | ianw: re zonefiles I think ns1 and ns2 only serve zones transfered to them by adns1 | 00:25 |
| clarkb | so they may not need "real" files on disk | 00:25 |
| *** wolverineav has quit IRC | 00:29 | |
| *** tbachman has joined #openstack-infra | 00:32 | |
| ianw | yeah, i think that's right, we can probably drop that section from the config files | 00:32 |
| ianw | - Fix crash for DS query when parent and child zones both configured | 01:06 |
| ianw | in nsd.conf and parent zone has not loaded properly. ... looks suspiciously like this | 01:06 |
| *** jamesmcarthur has joined #openstack-infra | 01:07 | |
| ianw | https://www.nlnetlabs.nl/bugs-script/show_bug.cgi?id=615 and here's a bug that looks weirdly like our not writing out zone files | 01:09 |
| openstack | www.nlnetlabs.nl bug 615 in NSD Code "NSD fails to write slave zone files" [Normal,Assigned] - Assigned to nsd-team | 01:09 |
| *** jamesmcarthur has quit IRC | 01:19 | |
| *** jamesmcarthur has joined #openstack-infra | 01:21 | |
| *** wolverineav has joined #openstack-infra | 01:26 | |
| *** jamesmcarthur has quit IRC | 01:30 | |
| ianw | https://www.nlnetlabs.nl/bugs-script/show_bug.cgi?id=4244 | 01:34 |
| openstack | www.nlnetlabs.nl bug 4244 in NSD Code "Segfault : add_rrset (... rrset=0x0) at query.c:717" [Normal,New] - Assigned to nsd-team | 01:34 |
| ianw | hopefully that helps ... now ... try and see what can do with ns1 :/ | 01:35 |
| *** jamesmcarthur has joined #openstack-infra | 01:42 | |
| *** bhavikdbavishi has joined #openstack-infra | 01:54 | |
| *** jamesmcarthur has quit IRC | 01:57 | |
| *** fatema__ has quit IRC | 02:00 | |
| *** yamamoto has joined #openstack-infra | 02:02 | |
| *** hongbin has joined #openstack-infra | 02:05 | |
| *** ricolin has joined #openstack-infra | 02:07 | |
| *** irclogbot_3 has quit IRC | 02:19 | |
| *** jamesmcarthur has joined #openstack-infra | 02:19 | |
| *** jamesmcarthur has quit IRC | 02:21 | |
| *** jamesmcarthur_ has joined #openstack-infra | 02:21 | |
| *** edmondsw has quit IRC | 02:22 | |
| *** jamesmcarthur_ has quit IRC | 02:30 | |
| *** jamesmcarthur has joined #openstack-infra | 02:34 | |
| *** wolverineav has quit IRC | 02:35 | |
| *** wolverineav has joined #openstack-infra | 02:38 | |
| *** jamesmcarthur has quit IRC | 02:39 | |
| *** wolverineav has quit IRC | 02:39 | |
| *** gregoryo has joined #openstack-infra | 02:43 | |
| tonyb | Any chance we can get another core review on https://review.openstack.org/648559 ? | 02:44 |
| *** apetrich has quit IRC | 02:58 | |
| *** yamamoto has quit IRC | 03:04 | |
| *** jamesmcarthur has joined #openstack-infra | 03:05 | |
| *** jamesmcarthur has quit IRC | 03:07 | |
| *** jamesmcarthur has joined #openstack-infra | 03:09 | |
| *** jamesmcarthur has quit IRC | 03:12 | |
| *** psachin has joined #openstack-infra | 03:17 | |
| ianw | ok, xfrd.state missing acme.opendev.org might be a red herring, it's only dumped on exit | 03:22 |
| *** ramishra has joined #openstack-infra | 03:24 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add openstack/freezer-tempest-plugin build job https://review.openstack.org/648559 | 03:29 |
| *** yamamoto has joined #openstack-infra | 03:35 | |
| *** lpetrut has joined #openstack-infra | 03:50 | |
| *** edmondsw has joined #openstack-infra | 04:02 | |
| *** udesale has joined #openstack-infra | 04:02 | |
| *** auristor has quit IRC | 04:06 | |
| *** ykarel has joined #openstack-infra | 04:08 | |
| *** auristor has joined #openstack-infra | 04:09 | |
| *** hongbin has quit IRC | 04:17 | |
| *** lpetrut has quit IRC | 04:19 | |
| openstackgerrit | Merged openstack-infra/zone-opendev.org master: Add DS record for acme.opendev.org https://review.openstack.org/648607 | 04:36 |
| openstackgerrit | Merged openstack-infra/zone-opendev.org master: Fix trailing dots for acme-challenge records https://review.openstack.org/648608 | 04:36 |
| *** jaosorior has joined #openstack-infra | 04:39 | |
| *** Lucas_Gray has joined #openstack-infra | 04:39 | |
| *** wolverineav has joined #openstack-infra | 04:40 | |
| *** wolverineav has quit IRC | 04:46 | |
| *** ykarel has quit IRC | 04:52 | |
| *** jamesmcarthur has joined #openstack-infra | 05:01 | |
| *** jamesmcarthur has quit IRC | 05:05 | |
| *** psachin has quit IRC | 05:09 | |
| *** psachin has joined #openstack-infra | 05:15 | |
| *** ykarel has joined #openstack-infra | 05:15 | |
| *** yamamoto has quit IRC | 05:16 | |
| *** yamamoto has joined #openstack-infra | 05:17 | |
| *** ianychoi has quit IRC | 05:23 | |
| *** ianychoi has joined #openstack-infra | 05:24 | |
| ianw | [Mon Apr 1 16:26:48 AEDT 2019] Cert success. | 05:27 |
| ianw | yay, we got a cert ... getting closer! | 05:27 |
| *** whoami-rajat has joined #openstack-infra | 05:31 | |
| *** chandankumar has joined #openstack-infra | 05:33 | |
| *** dpawlik has joined #openstack-infra | 05:38 | |
| *** quiquell has joined #openstack-infra | 05:42 | |
| *** jaosorior has quit IRC | 05:46 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul master: runner: add execute sub-command https://review.openstack.org/630944 | 05:46 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul master: zuul-runner: add command line interface https://review.openstack.org/644770 | 05:46 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul master: runner: add job parameters listing https://review.openstack.org/644795 | 05:46 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul master: Add depends-on support to frozen jobs API https://review.openstack.org/639022 | 05:46 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul master: zuul-runner: add support for depends-on https://review.openstack.org/632064 | 05:47 |
| *** jaosorior has joined #openstack-infra | 05:47 | |
| *** piotrowskim has joined #openstack-infra | 05:48 | |
| *** Lucas_Gray has quit IRC | 05:49 | |
| piotrowskim | https://www.irccloud.com/pastebin/Ikfsfjg6/ | 05:53 |
| *** kjackal has joined #openstack-infra | 05:57 | |
| *** psachin has quit IRC | 05:57 | |
| *** psachin has joined #openstack-infra | 06:05 | |
| *** slaweq has joined #openstack-infra | 06:19 | |
| *** janki has joined #openstack-infra | 06:21 | |
| *** pgaxatte has joined #openstack-infra | 06:28 | |
| *** kopecmartin|off is now known as kopecmartin | 06:30 | |
| *** Dobroslaw_afk is now known as Dobroslaw | 06:33 | |
| *** ccamacho has quit IRC | 06:53 | |
| *** gregoryo has quit IRC | 06:56 | |
| *** roman_g has joined #openstack-infra | 06:57 | |
| *** lpetrut has joined #openstack-infra | 06:57 | |
| *** rcernin has quit IRC | 06:58 | |
| *** pcaruana has joined #openstack-infra | 06:58 | |
| *** ginopc has joined #openstack-infra | 07:01 | |
| *** pcaruana has quit IRC | 07:02 | |
| *** pcaruana has joined #openstack-infra | 07:02 | |
| *** psachin has quit IRC | 07:08 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: executor: use node python path https://review.openstack.org/637339 | 07:10 |
| *** tosky has joined #openstack-infra | 07:13 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: letsencrypt support https://review.openstack.org/636759 | 07:13 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Add graphite01.opendev.org letsencrypt configuration https://review.openstack.org/648288 | 07:13 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Remove zonefile from nsd config https://review.openstack.org/648907 | 07:13 |
| *** psachin has joined #openstack-infra | 07:16 | |
| *** psachin has quit IRC | 07:16 | |
| *** quiquell is now known as quiquell|brb | 07:22 | |
| *** iurygregory has joined #openstack-infra | 07:22 | |
| *** ccamacho has joined #openstack-infra | 07:22 | |
| *** rpittau|afk is now known as rpittau | 07:24 | |
| *** ginux has joined #openstack-infra | 07:32 | |
| *** apetrich has joined #openstack-infra | 07:32 | |
| *** ginopc has quit IRC | 07:33 | |
| *** ginux is now known as ginopc | 07:33 | |
| *** ykarel is now known as ykarel|lunch | 07:35 | |
| *** e0ne has joined #openstack-infra | 07:40 | |
| *** quiquell|brb is now known as quiquell | 07:46 | |
| *** sshnaidm|off is now known as sshnaidm|rover | 07:48 | |
| *** lucasagomes has joined #openstack-infra | 07:52 | |
| *** jbadiapa has joined #openstack-infra | 07:54 | |
| *** jpena|off is now known as jpena | 07:57 | |
| openstackgerrit | YumengBao proposed openstack-infra/project-config master: Add api-ref-jobs for Cyborg https://review.openstack.org/648914 | 07:58 |
| *** jpich has joined #openstack-infra | 07:59 | |
| *** zbr is now known as zbr|pto | 08:08 | |
| *** e0ne has quit IRC | 08:11 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Use a more visible selection color https://review.openstack.org/648865 | 08:15 |
| *** e0ne has joined #openstack-infra | 08:16 | |
| *** ykarel|lunch is now known as ykarel | 08:19 | |
| *** wolverineav has joined #openstack-infra | 08:30 | |
| *** iurygregory has quit IRC | 08:32 | |
| *** iurygregory has joined #openstack-infra | 08:33 | |
| *** tkajinam has quit IRC | 08:34 | |
| *** wolverineav has quit IRC | 08:34 | |
| *** dtantsur|afk is now known as dtantsur | 08:35 | |
| *** derekh has joined #openstack-infra | 08:37 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Document how to run eslint locally https://review.openstack.org/648855 | 08:42 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: executor: add log_stream_port and log_stream_file settings https://review.openstack.org/535538 | 08:48 |
| *** e0ne has quit IRC | 08:50 | |
| *** e0ne has joined #openstack-infra | 08:59 | |
| *** adriancz has joined #openstack-infra | 09:02 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add support for checkbox and list parameters https://review.openstack.org/648661 | 09:03 |
| *** Lucas_Gray has joined #openstack-infra | 09:03 | |
| openstackgerrit | Daniel Mellado proposed openstack-infra/irc-meetings master: Update kuryr-kubernetes irc meeting time https://review.openstack.org/648927 | 09:04 |
| *** bobh has joined #openstack-infra | 09:06 | |
| *** xek has joined #openstack-infra | 09:10 | |
| *** bobh has quit IRC | 09:11 | |
| *** electrofelix has joined #openstack-infra | 09:14 | |
| *** otherwiseguy has quit IRC | 09:15 | |
| *** jrist has quit IRC | 09:15 | |
| *** mhu has quit IRC | 09:15 | |
| *** mhu has joined #openstack-infra | 09:16 | |
| *** jrist has joined #openstack-infra | 09:16 | |
| *** e0ne has quit IRC | 09:16 | |
| *** otherwiseguy has joined #openstack-infra | 09:17 | |
| *** e0ne has joined #openstack-infra | 09:35 | |
| *** ramishra has quit IRC | 09:40 | |
| *** gfidente has joined #openstack-infra | 09:40 | |
| *** ramishra has joined #openstack-infra | 09:43 | |
| *** ramishra_ has joined #openstack-infra | 09:48 | |
| *** gfidente has quit IRC | 09:50 | |
| *** ramishra has quit IRC | 09:50 | |
| *** roman_g has quit IRC | 09:54 | |
| *** roman_g has joined #openstack-infra | 09:57 | |
| *** yamamoto has quit IRC | 09:59 | |
| *** lucasagomes has quit IRC | 10:01 | |
| *** lucasagomes has joined #openstack-infra | 10:04 | |
| *** ramishra has joined #openstack-infra | 10:07 | |
| *** gfidente has joined #openstack-infra | 10:09 | |
| *** ramishra_ has quit IRC | 10:10 | |
| *** rnoriega has joined #openstack-infra | 10:22 | |
| *** sshnaidm|rover is now known as sshnaidm|rover|e | 10:22 | |
| rnoriega | hello guys! could you help me here? I'm core reviewer of networking-l2gw project, and when I try to push a new tag, I get this: | 10:23 |
| rnoriega | git push gerrit 14.0.0 | 10:23 |
| rnoriega | remote: Permission to openstack/networking-l2gw.git denied to oglok. | 10:23 |
| rnoriega | fatal: unable to access 'https://github.com/openstack/networking-l2gw/': The requested URL returned error: 403 | 10:23 |
| *** Lucas_Gray has quit IRC | 10:23 | |
| rnoriega | AJaeger, ^^ | 10:25 |
| *** yamamoto has joined #openstack-infra | 10:27 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: Proposed spec: tenant-scoped admin web API https://review.openstack.org/562321 | 10:29 |
| *** Lucas_Gray has joined #openstack-infra | 10:31 | |
| *** wolverineav has joined #openstack-infra | 10:31 | |
| *** yamamoto has quit IRC | 10:32 | |
| *** wolverineav has quit IRC | 10:35 | |
| *** tbachman has quit IRC | 10:37 | |
| *** ramishra has quit IRC | 10:38 | |
| *** ramishra has joined #openstack-infra | 10:40 | |
| *** yamamoto has joined #openstack-infra | 10:42 | |
| *** e0ne has quit IRC | 10:45 | |
| *** yamamoto has quit IRC | 10:46 | |
| *** yamamoto has joined #openstack-infra | 10:49 | |
| *** ykarel is now known as ykarel|afk | 10:53 | |
| *** efried has quit IRC | 10:54 | |
| zigo | ianw: Hi! Can you review this? https://review.openstack.org/#/c/645574/ | 10:56 |
| zigo | It should be ok now that we have Buster mirrored. | 10:56 |
| frickler | rnoriega: seems your git remote url is wrong, iiuc you should only be able to push your tags to review.openstack.org, not to github.com | 10:58 |
| rnoriega | frickler, yes, that's right. Thanks! it's Monday... xD xD | 10:58 |
| *** efried has joined #openstack-infra | 11:01 | |
| *** e0ne has joined #openstack-infra | 11:03 | |
| *** e0ne has quit IRC | 11:10 | |
| *** ykarel|afk is now known as ykarel | 11:12 | |
| *** bhavikdbavishi has quit IRC | 11:12 | |
| *** udesale has quit IRC | 11:22 | |
| *** EvilienM is now known as EmilienM | 11:27 | |
| *** jpena is now known as jpena|lunch | 11:35 | |
| *** rh-jelabarre has joined #openstack-infra | 11:41 | |
| *** tbachman has joined #openstack-infra | 11:45 | |
| *** sshnaidm|rover|e is now known as sshnaidm|rover | 11:46 | |
| *** weshay|rover is now known as weshay | 12:03 | |
| *** quiquell is now known as quiquell|lunch | 12:09 | |
| *** e0ne has joined #openstack-infra | 12:11 | |
| *** priteau has joined #openstack-infra | 12:11 | |
| *** trown|outtypewww is now known as trown | 12:12 | |
| *** rlandy has joined #openstack-infra | 12:14 | |
| *** apetrich has quit IRC | 12:16 | |
| *** rfolco has joined #openstack-infra | 12:16 | |
| *** yamamoto has quit IRC | 12:17 | |
| *** apetrich has joined #openstack-infra | 12:17 | |
| *** rcernin has joined #openstack-infra | 12:19 | |
| *** yamamoto has joined #openstack-infra | 12:19 | |
| *** ramishra has quit IRC | 12:25 | |
| *** bhavikdbavishi has joined #openstack-infra | 12:27 | |
| *** jpena|lunch is now known as jpena | 12:29 | |
| *** ramishra has joined #openstack-infra | 12:30 | |
| *** wolverineav has joined #openstack-infra | 12:32 | |
| *** markvoelker has quit IRC | 12:32 | |
| *** apetrich has quit IRC | 12:36 | |
| *** wolverineav has quit IRC | 12:36 | |
| *** e0ne has quit IRC | 12:38 | |
| *** quiquell|lunch is now known as quiquell | 12:43 | |
| *** nicolasbock has joined #openstack-infra | 12:45 | |
| *** nicolasbock has quit IRC | 12:47 | |
| *** yamamoto has quit IRC | 12:47 | |
| *** nicolasbock has joined #openstack-infra | 12:47 | |
| *** jamesmcarthur has joined #openstack-infra | 12:48 | |
| *** jamesmcarthur has quit IRC | 12:48 | |
| *** jamesmcarthur has joined #openstack-infra | 12:48 | |
| *** ricolin has quit IRC | 12:50 | |
| *** priteau has quit IRC | 12:51 | |
| *** priteau has joined #openstack-infra | 12:53 | |
| *** apetrich has joined #openstack-infra | 12:53 | |
| *** mriedem has joined #openstack-infra | 12:55 | |
| *** anteaya has joined #openstack-infra | 12:59 | |
| *** whoami-rajat has quit IRC | 13:00 | |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Update kuryr-kubernetes irc meeting time https://review.openstack.org/648927 | 13:04 |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: WIP - Pagure driver https://review.openstack.org/604404 | 13:04 |
| *** ykarel is now known as ykarel|afk | 13:05 | |
| *** whoami-rajat has joined #openstack-infra | 13:05 | |
| *** aaronsheffield has joined #openstack-infra | 13:08 | |
| *** bhavikdbavishi has quit IRC | 13:13 | |
| *** yamamoto has joined #openstack-infra | 13:14 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Delete files in dest that don't exist https://review.openstack.org/648815 | 13:18 |
| *** kgiusti has joined #openstack-infra | 13:19 | |
| *** lseki has joined #openstack-infra | 13:24 | |
| *** Lucas_Gray has quit IRC | 13:26 | |
| *** irclogbot_2 has joined #openstack-infra | 13:26 | |
| *** rcernin has quit IRC | 13:29 | |
| *** Lucas_Gray has joined #openstack-infra | 13:33 | |
| *** jroll has quit IRC | 13:39 | |
| *** eharney has joined #openstack-infra | 13:39 | |
| *** jroll has joined #openstack-infra | 13:39 | |
| *** ramishra has quit IRC | 13:41 | |
| *** jamesmcarthur has quit IRC | 13:43 | |
| *** ykarel|afk is now known as ykarel | 13:45 | |
| *** jcoufal has joined #openstack-infra | 13:53 | |
| *** ramishra has joined #openstack-infra | 13:56 | |
| *** tbachman has quit IRC | 13:57 | |
| *** jamesmcarthur has joined #openstack-infra | 13:58 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Do a ssh keyscan before running the git push mirror command https://review.openstack.org/649033 | 13:58 |
| openstackgerrit | Eric Fried proposed openstack-infra/elastic-recheck master: Add query for nova-live-migration bug 1822605 https://review.openstack.org/649034 | 13:58 |
| dmsimard | infra-root: ^ I think this is the last remaining thing to get the job working | 13:58 |
| openstack | bug 1822605 in OpenStack Compute (nova) "nova-live-migration fails 100% with "Multiple possible networks found, use a Network ID to be more specific"" [Critical,Triaged] https://launchpad.net/bugs/1822605 - Assigned to Matt Riedemann (mriedem) | 13:58 |
| *** mujahidali has joined #openstack-infra | 14:01 | |
| *** priteau has quit IRC | 14:03 | |
| *** mrhillsman_wkend is now known as mrhillsman | 14:04 | |
| openstackgerrit | Stephen Finucane proposed openstack-infra/zuul master: tox: Remove 'commands_pre' https://review.openstack.org/649038 | 14:05 |
| openstackgerrit | Stephen Finucane proposed openstack-infra/zuul master: tox: Change default 'install_command' https://review.openstack.org/649039 | 14:05 |
| openstackgerrit | Stephen Finucane proposed openstack-infra/zuul master: tox: Fix indentation https://review.openstack.org/649040 | 14:05 |
| openstackgerrit | Stephen Finucane proposed openstack-infra/zuul master: tox: Integrate tox-docker https://review.openstack.org/649041 | 14:05 |
| efried | is it possible to reply to a ML thread when I don't have the message in my client anymore? | 14:05 |
| *** e0ne has joined #openstack-infra | 14:06 | |
| stephenfin | efried: I've achieved that by (a) downloading archives and opening them up in mutt or (b) simply setting the subject and 'in-reply-to' header fields accordingly, if your client will let you do that | 14:06 |
| efried | stephenfin: in-reply-to...? | 14:07 |
| *** Lucas_Gray has quit IRC | 14:07 | |
| stephenfin | efried: https://www.jwz.org/doc/threading.html | 14:07 |
| stephenfin | Just set 'In-Reply-To: <message-id-of-original-mail>' | 14:08 |
| efried | okay, so I need to figure out the message ID. | 14:08 |
| stephenfin | Or just set the subject the same and rely on clients grouping emails with the same subject | 14:08 |
| efried | mm | 14:09 |
| * stephenfin knows way too much about this stuff on account of maintaining patchwork.ozlabs.org | 14:10 | |
| *** gfidente has quit IRC | 14:11 | |
| mnaser | infra-root: please keep in mind and let me know if you spot funky things in sjc1 | 14:13 |
| *** tbachman has joined #openstack-infra | 14:13 | |
| mnaser | it may or may not be running on beer | 14:14 |
| *** gfidente has joined #openstack-infra | 14:16 | |
| Shrews | mnaser: seeing launch failures for sjc1 in nodepool | 14:18 |
| mnaser | ehh | 14:18 |
| mnaser | grafana yesterday seemed ok | 14:18 |
| Shrews | mnaser: not sure how widespread it is, just looked at the most recent log entries so far | 14:19 |
| mnaser | yeah | 14:19 |
| mnaser | I see it in grafana now | 14:19 |
| mnaser | looks like they're they're not all failing | 14:19 |
| mnaser | OverQuota: VolumeSizeExceedsAvailableQuota: Requested volume or snapshot exceeds allowed gigabytes quota. Requested 80G, quota is 6144G and 6080G has been consumed. | 14:20 |
| *** tbachman has quit IRC | 14:20 | |
| mnaser | maybe leaked volumes, let me see | 14:20 |
| mnaser | I see 6 available volumes inside the openstackjenkins project | 14:22 |
| mnaser | deleting that might just stop those errors | 14:22 |
| mnaser | I won't do that, I'll defer to an infra-root to verify, or I can do it if I have infra telling me its ok :) | 14:22 |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Add query for nova-live-migration bug 1822605 https://review.openstack.org/649034 | 14:26 |
| openstack | bug 1822605 in OpenStack Compute (nova) "nova-live-migration fails 100% with "Multiple possible networks found, use a Network ID to be more specific"" [Critical,In progress] https://launchpad.net/bugs/1822605 - Assigned to Matt Riedemann (mriedem) | 14:26 |
| *** Lucas_Gray has joined #openstack-infra | 14:30 | |
| pabelanger | mnaser: yah, looks like a few leaked volumes | 14:32 |
| pabelanger | I can clean them up now | 14:32 |
| fungi | efried: stephenfin: not sure if you've already figured it out, but if you do a "show source" on the web archive page for the message you want to reply to you'll see there's a mailto href with a in-reply-to url parameter you can use, you just need to undo the % escaping (generally of the <, @ and > characters) | 14:33 |
| *** amansi26 has joined #openstack-infra | 14:37 | |
| pabelanger | mnaser: looks like a volume is stuck in deleting state: f4ec60a9-3461-4660-bf38-7b4b5429ede2 | 14:37 |
| *** ykarel is now known as ykarel|away | 14:37 | |
| pabelanger | mnaser: and a few stuck in creating | 14:38 |
| pabelanger | http://paste.openstack.org/show/748666/ | 14:38 |
| mnaser | pabelanger: could have been an inflight upgrade thing | 14:39 |
| *** derekh has quit IRC | 14:40 | |
| pabelanger | yah, unable to delete them. I think you'll have to | 14:40 |
| *** derekh has joined #openstack-infra | 14:40 | |
| *** wolverineav has joined #openstack-infra | 14:41 | |
| *** ykarel|away has quit IRC | 14:42 | |
| efried | fungi: Nice, thanks! | 14:44 |
| *** dpawlik has quit IRC | 14:44 | |
| amansi26 | I am facing a issue regarding ssh in my VM's. So whenever a VM is being created by nodepool, I am able to ping that VM but when I try to ssh in that VM, it throw: key-load-public-invalid-format. | 14:45 |
| amansi26 | Can someone help with this | 14:45 |
| *** wolverineav has quit IRC | 14:46 | |
| *** janki has quit IRC | 14:47 | |
| *** janki has joined #openstack-infra | 14:47 | |
| mordred | Shrews: do we need to track down a volume-leak thing? | 14:47 |
| *** roman_g has quit IRC | 14:53 | |
| openstackgerrit | Malek Karray proposed openstack-infra/storyboard-webclient master: Converting a search query into a browse https://review.openstack.org/644424 | 14:53 |
| *** roman_g has joined #openstack-infra | 14:53 | |
| *** dpawlik has joined #openstack-infra | 14:56 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add job to test upload-git-mirror on zuul-jobs https://review.openstack.org/648501 | 14:57 |
| Shrews | mordred: maybe, but... what's leaking them? | 14:59 |
| mordred | Shrews: yeah. I guess it's not us, right? because we're creating those vms for boot-from-volume with "please to delete the volume when the server deletes" flag yeah? | 15:00 |
| *** josephrsandoval has joined #openstack-infra | 15:00 | |
| *** dpawlik has quit IRC | 15:00 | |
| mnaser | Shrews, mordred: there hasn't been much leaks, but then again, this was a recently upgraded. cloud and in my experience there is a few "moments" where things can get weird | 15:01 |
| Shrews | mordred: if you mean nodepool, yes | 15:01 |
| mnaser | so I wouldn't blame it on leaks yet, but maybe it would be a nice-to-have to clean up after yourself | 15:01 |
| mordred | mnaser: ah - so maybe the leak happened during an upgrade just because of temporal weirdness? | 15:01 |
| mnaser | yep mordred | 15:02 |
| mnaser | because that cloud is running on beer now :> | 15:03 |
| mordred | \o/ | 15:03 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Don't create bindep venv if bindep_file is not found https://review.openstack.org/648833 | 15:04 |
| *** lpetrut has quit IRC | 15:04 | |
| *** josephrsandoval has quit IRC | 15:05 | |
| *** josephrsandoval has joined #openstack-infra | 15:06 | |
| *** michael-beaver has joined #openstack-infra | 15:07 | |
| *** priteau has joined #openstack-infra | 15:08 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Fixed bug on cloud assets https://review.openstack.org/649063 | 15:08 |
| *** owalsh has quit IRC | 15:11 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/storyboard master: Add a dedicated "lpimport" tox testenv https://review.openstack.org/649065 | 15:12 |
| *** josephrsandoval has quit IRC | 15:12 | |
| *** josephrsandoval has joined #openstack-infra | 15:13 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Fixed bug on cloud assets https://review.openstack.org/649063 | 15:13 |
| openstackgerrit | Merged openstack-infra/project-config master: Trove: use storyboard https://review.openstack.org/648320 | 15:13 |
| mnaser | looks like there's still errors according to grafana | 15:14 |
| mnaser | I wonder if cinder quotas got out of sync | 15:14 |
| *** dpawlik has joined #openstack-infra | 15:15 | |
| fungi | amansi26: when you say "when I try to ssh in that VM" do you mean you're manually trying to ssh to the node? or ansible on your zuul executor is trying to ssh to it and reporting that error? | 15:15 |
| *** ykarel|away has joined #openstack-infra | 15:15 | |
| *** ykarel|away is now known as ykarel | 15:15 | |
| *** bobh has joined #openstack-infra | 15:15 | |
| pabelanger | mnaser: I see more volumes in use then we have servers online | 15:17 |
| pabelanger | http://paste.openstack.org/show/748670/ | 15:17 |
| pabelanger | I am unsure where those servers are | 15:17 |
| pabelanger | I can unable to see them | 15:17 |
| fungi | amansi26: according to a quick web search that's ssh-agent complaining that it's unable to read at least one of the public keys corresponding to the private keys you have available | 15:18 |
| pabelanger | created_at: 2019-03-31T22:17:34.000000 | 15:18 |
| pabelanger | I think that was the window for upgrades? | 15:18 |
| amansi26 | fungi: Both. In addition to that when Zuul executor is trying and not able to ssh in that VM, it delete that VM and spawn a new VM. And it is happening for all the VM's. | 15:18 |
| mriedem | does anyone know if rax-dfw has their own pypi mirror? | 15:19 |
| *** bobh has quit IRC | 15:19 | |
| *** dpawlik has quit IRC | 15:19 | |
| mriedem | trying to figure out https://bugs.launchpad.net/devstack/+bug/1820892 which only hits on rax-dfw nodes | 15:19 |
| openstack | Launchpad bug 1820892 in devstack "Intermittent "Error starting thread.: ModuleNotFoundError: No module named 'etcd3gw'" in grenade-py3 jobs since March 14" [High,Fix released] - Assigned to Matt Riedemann (mriedem) | 15:19 |
| mriedem | and for certain packages, like libvirt-python, devstack is failing to grep a py3 classifier out of the remote pypi json response | 15:20 |
| mriedem | but when i run the same devstack code locally it's fine | 15:20 |
| *** quiquell is now known as quiquell|off | 15:20 | |
| fungi | amansi26: what version of zuul are you running? is this a new installation from scratch or one you upgraded? did it only recently stop working? | 15:23 |
| *** zhangfei has joined #openstack-infra | 15:23 | |
| clarkb | mriedem: we proxy and cache pypi now as its rate of growth for mirroring wasnt sustainable | 15:24 |
| *** e0ne has quit IRC | 15:24 | |
| clarkb | mriedem: this means each region has a different view of pypi depending on its cache state but indexes have a really short ttyl of like 5 minutesiirc | 15:24 |
| clarkb | so any discrepancies should go away in a reasonable amount of time | 15:25 |
| fungi | mriedem: clarkb: though we do also still build wheel caches, which seems to be where http://logs.openstack.org/79/636079/9/gate/grenade-py3/153ff2c/logs/grenade.sh.txt.gz#_2019-03-29_23_30_47_850 obtained its libvirt-python download | 15:25 |
| fungi | Downloading http://mirror.dfw.rax.openstack.org/wheel/ubuntu-18.04-x86_64/libvirt-python/libvirt_python-4.6.0-cp27-cp27mu-linux_x86_64.whl (442kB) | 15:25 |
| fungi | also worth noting, that job used a fairly ancient pip | 15:25 |
| fungi | "You are using pip version 9.0.3, however version 19.0.3 is available." | 15:25 |
| fungi | must be the distro package version | 15:25 |
| clarkb | ya devstack is still pinning pip | 15:26 |
| fungi | oh, i keep repressing all memory of that for some reason | 15:26 |
| clarkb | there are changes up to stop doing that but they havent all gotten in yet | 15:26 |
| *** owalsh has joined #openstack-infra | 15:26 | |
| clarkb | that was a chunk ofy friday reviewing | 15:27 |
| clarkb | frickler: I'll review the quoting fix shortly too | 15:27 |
| amansi26 | fungi: Recently it stopped working. Zuul version:2.6.0 | 15:27 |
| mriedem | hmm | 15:27 |
| mriedem | ok i'll run my local test script against http://mirror.dfw.rax.openstack.org/wheel/ubuntu-18.04-x86_64/libvirt-python/ | 15:27 |
| *** owalsh has quit IRC | 15:27 | |
| *** josephrsandoval has quit IRC | 15:27 | |
| fungi | amansi26: oh, so not zuul v3. are you using ansible or jenkins as your job runner? | 15:28 |
| fungi | mriedem: though that package should in theory be identical across all our providers since it's being served out of afs | 15:28 |
| amansi26 | fungi: jenkins-job-builder | 15:29 |
| openstackgerrit | Merged openstack-infra/storyboard-webclient master: Flawed logic when checking for invalid query https://review.openstack.org/645008 | 15:29 |
| fungi | amansi26: oh, so it's not zuul which is ssh'ing, it's the jenkins ssh slave plugin which reports that error message? or is nodepool giving that error when it tries to run the ready script? | 15:30 |
| openstackgerrit | Stephen Finucane proposed openstack-infra/project-config master: Retire oslosphinx https://review.openstack.org/649072 | 15:30 |
| mriedem | fungi: i'm not sure what's going on, but this pypi issue is definitely only on rax-dfw | 15:30 |
| fungi | mriedem: this leads me to wonder if there's something funky either with the afs/apache cache server we have in that region or with the local networking there | 15:31 |
| fungi | filesystems on it don't look full, no new kernel messages for nearly 3 weeks | 15:32 |
| amansi26 | fungi: Nodepool is throwing that error when it tries to run the ready script. | 15:33 |
| *** tbachman has joined #openstack-infra | 15:33 | |
| *** owalsh has joined #openstack-infra | 15:33 | |
| mriedem | looking at the py36 wheel http://mirror.dfw.rax.openstack.org/wheel/ubuntu-18.04-x86_64/libvirt-python/libvirt_python-4.6.0-cp36-cp36m-linux_x86_64.whl it's showing the py3 classifier | 15:34 |
| mriedem | Classifier: Programming Language :: Python :: 3 | 15:34 |
| mriedem | but the json check from pypi is what fails in devstack | 15:34 |
| mriedem | so i'm wondering where that pypi curl is going to? | 15:34 |
| fungi | mriedem: oh, you're hitting the pypi json metadata api | 15:34 |
| mriedem | right | 15:34 |
| fungi | we don't cache/mirror that at all | 15:35 |
| mriedem | this function https://github.com/openstack-dev/devstack/blob/master/inc/python#L103 | 15:35 |
| mriedem | oh ok | 15:35 |
| amansi26 | fungi:Error from /var/log/nodepool/nodepool.log | 15:35 |
| amansi26 | 2019-04-01 11:33:14,028 ERROR nodepool.utils: Exception while testing ssh access: | 15:35 |
| amansi26 | Traceback (most recent call last): | 15:35 |
| amansi26 | File "/usr/local/lib/python2.7/dist-packages/nodepool/nodeutils.py", line 55, in ssh_connect | 15:35 |
| amansi26 | client = SSHClient(ip, username, **connect_kwargs) | 15:35 |
| amansi26 | File "/usr/local/lib/python2.7/dist-packages/nodepool/sshclient.py", line 30, in __init__ | 15:35 |
| amansi26 | key_filename=key_filename) | 15:35 |
| amansi26 | File "/usr/local/lib/python2.7/dist-packages/paramiko/client.py", line 304, in connect | 15:35 |
| amansi26 | retry_on_signal(lambda: sock.connect(addr)) | 15:35 |
| amansi26 | File "/usr/local/lib/python2.7/dist-packages/paramiko/util.py", line 270, in retry_on_signal | 15:35 |
| amansi26 | return function() | 15:35 |
| amansi26 | File "/usr/local/lib/python2.7/dist-packages/paramiko/client.py", line 304, in <lambda> | 15:35 |
| amansi26 | retry_on_signal(lambda: sock.connect(addr)) | 15:35 |
| amansi26 | File "/usr/lib/python2.7/socket.py", line 228, in meth | 15:35 |
| amansi26 | return getattr(self._sock,name)(*args) | 15:35 |
| amansi26 | error: [Errno 110] Connection timed out | 15:35 |
| clarkb | fwiw the wheel fungi linked was a python2 wheel and maybe the wheel build process removes python3 trove entries when its python2 only? | 15:35 |
| clarkb | amansi26: please use a paste service to make large pastes | 15:35 |
| fungi | amansi26: in the future please don't spam the channel with errors, use something like http://paste.openstack.org/ | 15:35 |
| clarkb | amansi26: http://paste.openstack.org is one we run but there are a number out there | 15:35 |
| mriedem | clarkb: yeah devstack uses the py2 wheel after it fails to find the py3 classifier in pypi for the package | 15:36 |
| mriedem | https://github.com/openstack-dev/devstack/blob/master/inc/python#L300 | 15:36 |
| mriedem | i'm running check_python3_support_for_package_remote in a little test script locally and it has no problem finding the python 3 classifier for libvirt-python | 15:36 |
| fungi | mriedem: so i think that api ends up going through pypi.org's fastly cdn. perhaps there's something broken with one of the cdn endpoints nearest that provider/region | 15:37 |
| *** janki has quit IRC | 15:37 | |
| fungi | mriedem: if you can provide a copy of your test script i can try running it from a machine in rax-dfw and see what it does | 15:38 |
| mriedem | fungi: https://bugs.launchpad.net/devstack/+bug/1820892/comments/14 | 15:38 |
| openstack | Launchpad bug 1820892 in devstack "Intermittent "Error starting thread.: ModuleNotFoundError: No module named 'etcd3gw'" in grenade-py3 jobs since March 14" [High,Fix released] - Assigned to Matt Riedemann (mriedem) | 15:38 |
| fungi | thanks | 15:38 |
| mriedem | oh but you'd need devstack sourced, hold on | 15:38 |
| amansi26 | fungi: Sure. I was not aware of that. | 15:38 |
| mriedem | fungi: http://paste.openstack.org/show/748672/ | 15:39 |
| fungi | amansi26: "error: [Errno 110] Connection timed out" is nothing like the "key-load-public-invalid-format" error you originally mentioned | 15:39 |
| *** ykarel is now known as ykarel|away | 15:40 | |
| amansi26 | fungi:I tired login manually using the key and got that error. | 15:41 |
| fungi | amansi26: yeah, i don't think those two errors are likely related in any way | 15:43 |
| *** e0ne has joined #openstack-infra | 15:44 | |
| fungi | amansi26: the "key-load-public-invalid-format" error is probably a mistake in how you're running the command yourself, or in how you generated the key(s) you're using when you ran that command, or because of some other garbage in your ~/.ssh directory | 15:44 |
| fungi | amansi26: the ssh timeout error nodepool is reporting could be for a number of reasons, but there's a good chance it's due to something which has changed with the networking between your nodepool server and wherever your nodes are being booted | 15:45 |
| *** Wryhder has joined #openstack-infra | 15:46 | |
| clarkb | fungi: https://review.openstack.org/#/c/561597/16 is the stack that will update pip in devstack jobs | 15:46 |
| fungi | mriedem: running that script locally on the mirror.dfw.rax.o.o server (out of convenience) it consistently reports "Found package in pypi supports py3" | 15:46 |
| *** e0ne has quit IRC | 15:47 | |
| *** Lucas_Gray has quit IRC | 15:47 | |
| fungi | oh, hah, i should pass it a package name, looks like | 15:47 |
| mriedem | no it's hard-coded | 15:47 |
| fungi | no, nevermind, it's in there right | 15:47 |
| *** Wryhder is now known as Lucas_Gray | 15:47 | |
| mriedem | hmm ok | 15:47 |
| *** tbachman has quit IRC | 15:47 | |
| clarkb | cmurphy: re topic:puppet-4 it would probably be good to avoid upgrading puppet on any remaining trusty nodes (like lists.o.o) beacuse I've already sorted out how to fix the apt setup for puppet-3 hosts after an in place trusty to xenial upgrade but have not tested with puppet-4 | 15:48 |
| fungi | mriedem: so what would be the behavior of the code in devstack if https://pypi.python.org/pypi/libvirt-python/json occasionally came back truncated? | 15:48 |
| fungi | (or empty) | 15:48 |
| *** priteau has quit IRC | 15:48 | |
| clarkb | cmurphy: its probably not a major issue and if we get through good xenial candidates before done with trusty upgrades we can figure it out | 15:48 |
| mriedem | we'd fallback to py2 | 15:48 |
| clarkb | cmurphy: but until the xenial nodes are done I'd prefer we stick to those. | 15:48 |
| *** josephrsandoval has joined #openstack-infra | 15:49 | |
| geguileo | Trivial patch adding publish-to-pypi has one +2, anybody around with a minute to review and approve? https://review.openstack.org/#/c/643542/ | 15:49 |
| fungi | mriedem: so just to confirm, if we got incomplete data from that pypi json metadata api query on occasion, that could explain the behavior exhibited in that bug? | 15:49 |
| mriedem | fungi: i think so yeah | 15:49 |
| mriedem | b/c we're just doing a grep | 15:49 |
| *** pgaxatte has quit IRC | 15:49 | |
| clarkb | speaking of infra-root and other infraians I am going to work on sending test email through the test xenial lists server I booted once I get caught up after being out most of last week | 15:49 |
| fungi | mriedem: rather than actually trying to parse the json | 15:49 |
| clarkb | geguileo: I'll look | 15:49 |
| geguileo | clarkb: thanks! | 15:50 |
| mriedem | fungi: correct | 15:50 |
| mnaser | pabelanger: I will check those | 15:50 |
| mriedem | https://github.com/openstack-dev/devstack/blob/master/inc/python#L103 | 15:50 |
| * clarkb finds breakfast | 15:50 | |
| openstackgerrit | Stephen Finucane proposed openstack-infra/project-config master: Retire cloud-init https://review.openstack.org/649082 | 15:51 |
| amansi26 | fungi:Our CI is using pre generated ssh-keys to login into the VM's and I thought that the key might got corrupted. So, I tried login manually using the ssh-key, but got the first error. | 15:51 |
| amansi26 | then I tried login to nodepool user and I was able to ssh to VM and suddenly got disconnected. When I looked into the nodepool logs, I got the error pasted(mistakenly here). | 15:51 |
| geguileo | clarkb: thanks! :-) | 15:51 |
| mriedem | fungi: so you're thinking maybe curl is failing on some network error? | 15:52 |
| mriedem | it is using -s so any error would be surpressed | 15:52 |
| fungi | mriedem: or one of the nearby fastly cdn endpoints is responding with an incomplete payload | 15:52 |
| *** josephrsandoval has quit IRC | 15:52 | |
| openstackgerrit | Merged openstack-infra/zuul-sphinx master: Publish docs on release https://review.openstack.org/648779 | 15:56 |
| *** wolverineav has joined #openstack-infra | 16:00 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add cinderlib publish-to-pypi job https://review.openstack.org/643542 | 16:01 |
| openstackgerrit | Merged opendev/base-jobs master: Add opendev-publish-tox-docs https://review.openstack.org/648775 | 16:03 |
| *** ginopc has quit IRC | 16:03 | |
| *** lucasagomes has quit IRC | 16:04 | |
| *** wolverineav has quit IRC | 16:04 | |
| fungi | amansi26: i suggest you start by looking into what might have changed when nodepool ceased working for you | 16:05 |
| fungi | how long has it been acting this way? did you upgrade anything around the same time? | 16:05 |
| mriedem | fungi: ok i'll put up a debug patch for devstack and see if we can root this out a bit | 16:05 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: WIP: Fix for image build leaks https://review.openstack.org/647599 | 16:05 |
| clarkb | zigo: more pip updates https://github.com/pypa/pip/pull/6367#issuecomment-478463890 is suggested as proper long term fix for this. Any chance you are able to check it on debian? | 16:05 |
| *** dtantsur is now known as dtantsur|afk | 16:07 | |
| amansi26 | fungi: Not updated anything, just did a CI-redeploy. CI redeploy usaullay does not give me this error. | 16:08 |
| openstackgerrit | Merged openstack-infra/zuul master: Fix typos https://review.openstack.org/648368 | 16:08 |
| *** tbachman has joined #openstack-infra | 16:09 | |
| fungi | amansi26: what is a "ci-redeploy" and what does it di? | 16:10 |
| fungi | er, what does it do? | 16:10 |
| *** rpittau is now known as rpittau|afk | 16:13 | |
| *** Lucas_Gray has quit IRC | 16:13 | |
| rajinir | Hi all, are there any instructions to switch the thirdparty CI from Xenial to Bionic? | 16:14 |
| amansi26 | fungi: It tear downs and restack the devstack on controller node and, delete the config and readd them for zuul, jenkins and nodepool. I have not changed any config for zuul, jenkins or nodepool. | 16:15 |
| clarkb | rajinir: I don't think so. We don't prescribe test platform for third party CI | 16:17 |
| fungi | amansi26: is the devstack controller node what nodepool is trying to ssh into? (or do you even use nodepool in your system? sounds like you have static pre-built jenkins slaves instead?) | 16:18 |
| *** zhangfei has quit IRC | 16:20 | |
| rajinir | clarb: any tips on the config changes that are required? | 16:23 |
| clarkb | rajinir: I think for our jobs we updated base jobs to do it | 16:25 |
| amansi26 | fungi: Slave nodes are spawned using devstack only and nodepool is trying to ssh into the VM spawned by devstack. | 16:25 |
| clarkb | rajinir: so devstack-base or whatever that lowest common denominator devstack base job updated its nodeset to bionic | 16:25 |
| rajinir | clarkb: thanks will checkout | 16:25 |
| clarkb | rajinir: this allows child jobs to override if necessary but by default they all run on the newer distro | 16:25 |
| clarkb | rajinir: then for other jobs we ended up setting bionic as base node type in our global base job | 16:26 |
| fungi | clarkb: rajinir: for the most part, yes, we changed the nodeset in the base jobs and then set up variants of common parent jobs to continue using ubuntu-xenial on branches prior to stable/stein | 16:26 |
| *** michael-beaver has quit IRC | 16:26 | |
| mnaser | hmm | 16:26 |
| mnaser | looks like still seeing some weirdness with nodepool | 16:26 |
| *** michael-beaver has joined #openstack-infra | 16:27 | |
| *** wolverineav has joined #openstack-infra | 16:27 | |
| *** mujahidali has quit IRC | 16:27 | |
| stephenfin | clarkb, fungi: Do you think it would be possible to issue a new release of git-review? There's a lot of stuff piled up there | 16:27 |
| dmsimard | Need reviews on https://review.openstack.org/#/c/649033/ to hopefully fix the last outstanding issue with the new upload-git-mirror job | 16:28 |
| stephenfin | Including https://review.openstack.org/#/c/610573/ which is killing me slowly downstream | 16:28 |
| mnaser | 45 volumes in-use (aka 3600gb used), 51 instances in nova (including error etc | 16:28 |
| *** gfidente has quit IRC | 16:28 | |
| rajinir | fungi: clarkb: ack. thanks | 16:28 |
| mnaser | I think the quota is out of sync | 16:29 |
| fungi | amansi26: oh, got it. so anyway if nodepool started raising | 16:29 |
| fungi | "error: [Errno 110] Connection timed out" shortly after you redeployed the cloud it's talking to | 16:29 |
| clarkb | stephenfin: fungi maybe we do a 1.28.0a1 tag then some of us can test irc and if happy retag as 1.28.0 ? | 16:30 |
| fungi | amansi26: then odds are the virtual machines being launched in that devstack environment aren't reachable from your nodepool server | 16:30 |
| *** kopecmartin is now known as kopecmartin|off | 16:30 | |
| stephenfin | clarkb: No arguments from me | 16:30 |
| fungi | clarkb: sounds fine, sure | 16:30 |
| clarkb | ok I'll dig out my key shortly | 16:30 |
| stephenfin | clarkb: I was half way though an email to the Fedora package maintainer before realizing we'd made no releases in a while | 16:30 |
| fungi | clarkb: except it would be 1.28.0.0a1 | 16:31 |
| clarkb | fungi: yup good point | 16:31 |
| stephenfin | clarkb: Thanks :) Let me know if there's anything I can help with | 16:31 |
| *** hwoarang has quit IRC | 16:31 | |
| *** wolverineav has quit IRC | 16:32 | |
| *** dikonoor has joined #openstack-infra | 16:34 | |
| mnaser | ok it seems that cinder's stores quota usage in the db | 16:34 |
| amansi26 | fungi: But the thing is the virtual machine is pingable but as soon as nodepool found that it's not able to connect to vm it delete the VM. | 16:34 |
| mnaser | and when things choked up I guess they got messed up and now result is http://paste.openstack.org/show/748678/ | 16:35 |
| openstackgerrit | Riju Khatri proposed openstack-infra/storyboard master: Search bars handle queries as if they have a * at the end https://review.openstack.org/644487 | 16:35 |
| fungi | amansi26: yes, if nodepool can't connect to an instance it assumes that instance is broken in some way and deletes it to free up quota for launching more | 16:35 |
| fungi | amansi26: is it possible something has changed with the security groups in your devstack environment so that it's no longer allowing ssh from the nodepool server? | 16:36 |
| *** hwoarang has joined #openstack-infra | 16:36 | |
| mnaser | infra-root: I believe all the issues in sjc1 should now be cleared up, looks like cinder quota usage might be gotten messed up | 16:37 |
| clarkb | stephenfin: fungi 1.28.0.0a1 is pushed | 16:37 |
| clarkb | mnaser: thanks | 16:37 |
| * fungi installs | 16:38 | |
| fungi | oh, need to wait for the release jobs to run ;) | 16:39 |
| clarkb | fungi: it may not have hit pypi yet fwiw | 16:39 |
| fungi | yeah, just realized that after i ended up with git-review 1.27.1.0a1 | 16:39 |
| fungi | which, by the way, we pushed back in october and then never tagged 1.27.1 | 16:40 |
| clarkb | ya | 16:41 |
| fungi | oh well. here's to our never-ending stream of distractions! | 16:41 |
| amansi26 | fungi: No, nothing has been changed in the security groups in our devstack environment. | 16:41 |
| fungi | amansi26: you said you redeployed it though. do you deploy from the same code base every time? | 16:41 |
| fungi | or do you use newer versions of devstack, nova, neutron, and so on when you redeploy? | 16:42 |
| *** ramishra has quit IRC | 16:42 | |
| openstackgerrit | Riju Khatri proposed openstack-infra/storyboard master: Projects can be browsed using stories https://review.openstack.org/645169 | 16:42 |
| fungi | amansi26: for example, there was a recent bug fixed in nova which causes it to no longer fail open when faced with invalid security group rules: https://security.openstack.org/ossa/OSSA-2019-001.html | 16:44 |
| fungi | er, i mean in neutron | 16:44 |
| *** wolverineav has joined #openstack-infra | 16:44 | |
| *** e0ne has joined #openstack-infra | 16:45 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add link to builds in job page https://review.openstack.org/648864 | 16:48 |
| clarkb | fungi: stephenfin you can update to 1.28.0.0a1 now | 16:49 |
| *** wolverineav has quit IRC | 16:49 | |
| clarkb | (I just did | 16:49 |
| *** trown is now known as trown|lunch | 16:49 | |
| *** wolverin_ has joined #openstack-infra | 16:49 | |
| stephenfin | clarkb: Thanks | 16:49 |
| fungi | Successfully installed git-review-1.28.0.0a1 | 16:50 |
| *** wolverin_ has quit IRC | 16:51 | |
| *** wolverineav has joined #openstack-infra | 16:52 | |
| *** jpich has quit IRC | 16:53 | |
| *** wolverineav has quit IRC | 16:53 | |
| *** wolverineav has joined #openstack-infra | 16:53 | |
| openstackgerrit | Merged openstack-infra/zuul master: Fix slightly smaller font of in progress jobs https://review.openstack.org/648827 | 16:53 |
| amansi26 | fungi:I am not sure if this has impacted our CI. Can you please help me identifying if this is the reason? | 16:55 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Use xterm.js for live log streaming https://review.openstack.org/648838 | 16:56 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add search function to web streaming https://review.openstack.org/648862 | 16:56 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Use a more visible selection color https://review.openstack.org/648865 | 16:56 |
| fungi | amansi26: i think we're verging far off-topic for the community infra services channel, but i recommend checking whether iptables on your hypervisor hosts is blocking ssh from your nodepool server | 16:57 |
| amansi26 | fungi: Sure, I will check that, thank you | 17:00 |
| pabelanger | clarkb: pip question, how does one usually handle setup.py dependencies? for example, because you call setup(), you need another library to do something. I don't think there is a way to do that today in pip right? you need to do a 2 stage install, iirc | 17:01 |
| clarkb | pabelanger: setuptools has setup requires | 17:01 |
| clarkb | pabelanger: it bootstraps itself using easy_install iirc | 17:01 |
| clarkb | pip may pull them out itself now though | 17:02 |
| clarkb | this is how pbr gets installed | 17:02 |
| *** derekh has quit IRC | 17:02 | |
| pabelanger | great, thanks! looking | 17:03 |
| openstackgerrit | Merged openstack-infra/zuul master: Resolve todo after stream.html to stream renaming https://review.openstack.org/648828 | 17:03 |
| *** jpena is now known as jpena|off | 17:04 | |
| fungi | pabelanger: for example, https://opendev.org/openstack-infra/bindep/src/branch/master/setup.py#L28 | 17:06 |
| fungi | setup_requires is just a list of package names | 17:06 |
| fungi | that's for specifying install-time dependencies (as opposed to run-time dependencies) | 17:06 |
| *** bobh has joined #openstack-infra | 17:15 | |
| *** amansi26 has quit IRC | 17:15 | |
| pabelanger | fungi: clarkb: okay, thanks. that might work, now to figure out how to refresh pythonpath on running script. This isn't openstack specifc, some python app I am using with a weird chicken/egg dependency thing | 17:16 |
| *** ykarel|away has quit IRC | 17:20 | |
| fungi | pabelanger: not sure what you mean. you need to reimport changed modules or something else? | 17:20 |
| openstackgerrit | Merged openstack-dev/pbr master: Support provides_extra metadata https://review.openstack.org/597612 | 17:20 |
| openstackgerrit | Merged openstack-dev/pbr master: Typo fix: s/extract_mesages/extract_messages/ https://review.openstack.org/644942 | 17:20 |
| pabelanger | fungi: so, setup.py is already running, using easy_install() works, but if next line is import foo, from easy_install() python cannot find module. I am assuming this is because pythonpath for modules did refresh | 17:21 |
| pabelanger | I have no idea, if what I am asking is even possible | 17:21 |
| fungi | not sure why you'd have to update the pythonpath for that | 17:25 |
| dulek | Hey guys, can you freeze me a gate VM? We're dealing with some connectivity issues in kuryr-kubernetes gates. | 17:28 |
| dulek | https://review.openstack.org/#/c/626609 - you can use this patch. | 17:28 |
| dulek | And kuryr-kubernetes-tempest job will be fine. | 17:28 |
| fungi | dulek: it doesn't look like kuryr-kubernetes-tempest is run for that change | 17:30 |
| dulek | fungi: It's a bit old run… Okay, this will do as well: https://review.openstack.org/#/c/648497/ | 17:33 |
| fungi | oh, got it, the job's been added in the past couple months? | 17:33 |
| dulek | fungi: More like renamed, but yes, in the last 2 months. | 17:35 |
| *** e0ne has quit IRC | 17:35 | |
| fungi | okay, if you recheck 648497 and let me know when the kuryr-kubernetes-tempest job for it completes, i can add your ssh key to the held node(s) | 17:35 |
| openstackgerrit | Merged openstack-infra/zuul master: tox: Remove 'commands_pre' https://review.openstack.org/649038 | 17:36 |
| openstackgerrit | Merged openstack-infra/zuul master: tox: Change default 'install_command' https://review.openstack.org/649039 | 17:36 |
| dulek | fungi: Okay, will ping you. | 17:37 |
| *** zigo has quit IRC | 17:37 | |
| *** wolverineav has quit IRC | 17:39 | |
| *** wolverineav has joined #openstack-infra | 17:40 | |
| pabelanger | fungi: clarkb: thanks for the pointers, https://github.com/borgbackup/borg/pull/4484 is what I came up with | 17:41 |
| clarkb | pabelanger: hrm my guess is that cython is weird and requires the reload but generally you shouldn't need that | 17:43 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Updated Member/Speaker Serializer https://review.openstack.org/649145 | 17:43 |
| *** wolverin_ has joined #openstack-infra | 17:43 | |
| clarkb | pabelanger: also fwiw I use distro packaged borg and it works great. Ubuntu xenials is new enough to be past that upgrade trickyness period | 17:43 |
| *** yamamoto has quit IRC | 17:44 | |
| *** wolverineav has quit IRC | 17:44 | |
| fungi | strange that you would need to reload site unless something had already imported it earlier | 17:44 |
| clarkb | I'm prepping the infra meeting agenda now. Anything I missed last week we should put on the agenda? | 17:45 |
| *** michael-beaver has left #openstack-infra | 17:45 | |
| *** wolverin_ has quit IRC | 17:47 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Updated Member/Speaker Serializer https://review.openstack.org/649145 | 17:47 |
| fungi | clarkb: nothing springs to mind, no | 17:48 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Updated Member/Speaker Serializer https://review.openstack.org/649145 | 17:49 |
| pabelanger | clarkb: yah, been using it from pip for last few months, could give distro version ago also | 17:49 |
| fungi | tried borgmatic? that looks useful | 17:51 |
| fungi | https://torsion.org/borgmatic/ | 17:52 |
| clarkb | fungi: I just used the example shell script in borgs docs | 17:52 |
| clarkb | and added a borgrc so that I can do manual operations more easily | 17:53 |
| fungi | the automated backup validation and pruning features are compelling | 17:53 |
| clarkb | fungi: also fuse mounts for recovery | 17:53 |
| clarkb | and deduplication | 17:53 |
| clarkb | and append only if you want | 17:53 |
| clarkb | I've been really happy with it so far | 17:53 |
| fungi | well, those are features of borg itself | 17:53 |
| clarkb | well so are the other features :P | 17:54 |
| *** bobh has quit IRC | 17:54 | |
| clarkb | you just have to use the script in the docs | 17:54 |
| clarkb | though I think I disabled auto validation because it is expensive and I have slow cheap setup | 17:55 |
| fungi | ahh | 17:55 |
| fungi | do you backup to a remote location? | 17:55 |
| clarkb | fungi: yup I bought a raspi and attached a usb3 hdd to it and stuck it at my brothers house. We have the same ISP and are one router hop away from each other so I get full 100mbps duplex between us. I can also go grab the hdd and plug it in locally for recovery if necessary. The downside is a sufficiently large meteor would get my backup site too | 17:57 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: CloudAssest hotfix https://review.openstack.org/649150 | 17:57 |
| *** e0ne has joined #openstack-infra | 17:57 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: CloudAssest hotfix https://review.openstack.org/649150 | 17:57 |
| clarkb | re Zanata any idea if an official statement will be made upstream? | 17:58 |
| clarkb | if you poke around teh website and the mailing list there isn't any strong indication that the progress is idle | 17:59 |
| fungi | official as in an announcement it's being abandoned? | 17:59 |
| clarkb | https://www.redhat.com/archives/zanata-devel/2019-February/msg00000.html asks the question with no answer | 17:59 |
| clarkb | fungi: ya | 17:59 |
| clarkb | and I wonder what fedora is going to do about translations now? | 17:59 |
| clarkb | just thinking an official statement would help with planning | 18:00 |
| fungi | yeah, i see the messages from september about something happening with red hat's brisbane office, and then the list goes silent other than a bit of spam which slipped through in november | 18:02 |
| *** trown|lunch is now known as trown | 18:04 | |
| clarkb | we probably want to talk to fedora about their plans though as I imagine they have a bit more inside baseball info | 18:04 |
| clarkb | but their wiki still refers to zanata now | 18:04 |
| pabelanger | fungi: yah, borgmatic is nice too, using that also | 18:05 |
| openstackgerrit | Merged openstack-infra/zuul master: web: Add '/' API route https://review.openstack.org/648399 | 18:05 |
| *** wolverineav has joined #openstack-infra | 18:05 | |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: Rework docs jobs to be consumable by tenants https://review.openstack.org/649152 | 18:06 |
| *** dikonoor has quit IRC | 18:06 | |
| mordred | clarkb: I'll send a quick email around internally too - see what I can learn | 18:07 |
| chandankumar | AJaeger: fungi dmsimard please have a look at this review https://review.openstack.org/#/c/647667/ when free, thanks! | 18:07 |
| clarkb | mordred: fwiw sounds like ianw did some digging already (may want to sync up?) | 18:07 |
| mordred | oh - yeah? cool | 18:07 |
| mordred | clarkb: so - does that mean that we basically know it's been abandoned, but there isn't any public statement? | 18:08 |
| fungi | mordred: more like we know the original team aren't around and it's been radio silence on the usual channels | 18:09 |
| mordred | nod | 18:09 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Fix on cloudassets bucket container name https://review.openstack.org/649153 | 18:09 |
| corvus | so i think the question is -- since it looks like it's abandoned and we assume it is, what are fedora's plans for translations since they were using it previously as well? | 18:10 |
| fungi | i concur | 18:10 |
| clarkb | corvus: ++ | 18:10 |
| mordred | ++ | 18:10 |
| clarkb | mordred: from ianw's most recent email on the infra list there is no current support or planned future development | 18:10 |
| fungi | and we're past the 6 month mark since the original devs said they were leaving | 18:10 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Fix on cloudassets bucket container name https://review.openstack.org/649153 | 18:10 |
| *** electrofelix has quit IRC | 18:12 | |
| corvus | clarkb, fungi, mordred: i'm taking a stab at refactoring the new opendev docs jobs so that any opendev tenant can use them without alteration: can you look at https://review.openstack.org/649152 when you have a minute? when that lands, i'll rework the zuul tenant jobs and test all that out again. | 18:12 |
| corvus | it's a sort of greatest common denominator of what openstack, opendev, and zuul are doing with docs publication | 18:12 |
| clarkb | yup I'll add it to my queue | 18:13 |
| corvus | hopefully if we get that worked out, we can make new starlingx, airship, etc tenants with more reusable components | 18:13 |
| *** jamesmcarthur has quit IRC | 18:15 | |
| *** wolverineav has quit IRC | 18:16 | |
| *** wolverineav has joined #openstack-infra | 18:17 | |
| *** yamamoto has joined #openstack-infra | 18:19 | |
| *** wolverineav has quit IRC | 18:22 | |
| *** wolverineav has joined #openstack-infra | 18:23 | |
| dmsimard | fungi, clarkb, corvus: still trying to land everything for upload-git-mirror, https://review.openstack.org/649033 should hopefully be the last one before I can send an email out. Would appreciate a review when you have a chance <3 | 18:23 |
| clarkb | corvus: reviewed I think there are some bugs in the condition checking | 18:23 |
| clarkb | dmsimard: as an alternative do we think the job should be given the host key so that we verify we are pushing to the correct destination? | 18:24 |
| dmsimard | clarkb: tbh I frowned a bit at the keyscan because it's not much better than ignoring known hosts (considering we scan right before doing the push..) | 18:25 |
| clarkb | right, and the host key data is public info that the job coudl set | 18:26 |
| *** yamamoto has quit IRC | 18:26 | |
| *** e0ne has quit IRC | 18:28 | |
| *** jonher has quit IRC | 18:28 | |
| *** jonher has joined #openstack-infra | 18:29 | |
| clarkb | that said I'm not sure pushing public git repo data through a mitm attack is something that would be a problem? | 18:30 |
| clarkb | I suppose they could poison the remote | 18:30 |
| clarkb | which would be an issue | 18:30 |
| dmsimard | I just want the thing to work for now :( | 18:32 |
| fungi | we could just embed a literal copy of the host key, would be more robust and require fewer network calls | 18:33 |
| clarkb | fungi: ya that is what I mean | 18:33 |
| clarkb | set it as a job var | 18:33 |
| dmsimard | What if the host key is expected to change ? | 18:34 |
| clarkb | you would have to update the job to include the new key | 18:35 |
| clarkb | I don't expect that to be a common phenomenon though | 18:35 |
| dmsimard | clarkb: for the integration tests for that role, we currently push to a repository over ssh but it's on localhost | 18:36 |
| dmsimard | so we do a ssh keyscan prior to running the role | 18:36 |
| dmsimard | I guess we're allowed to cheat on tests :p | 18:37 |
| dmsimard | Could it be optional ? For example, if there's no host key provided, we do the keyscan. | 18:40 |
| clarkb | I worry that people would rarely set it in that case :/ | 18:41 |
| dmsimard | yeah ... | 18:42 |
| dmsimard | ok, let me see what it would look like | 18:42 |
| dulek | fungi: Okay, the kuryr-kubernetes-tempest job on https://review.openstack.org/648497 failed. | 18:42 |
| dulek | fungi: Here's my SSH key: https://github.com/dulek.keys | 18:43 |
| fungi | um... wow | 18:45 |
| fungi | oh, i get it | 18:45 |
| fungi | i didn't realize we'd turned limestone back up | 18:46 |
| fungi | dulek: do you have ipv6 connectivity? this particular job node is not globally reachable over ipv4 due to addressing limitations in that provider | 18:46 |
| clarkb | fungi: you might be able to create a floating ip and attach it | 18:47 |
| clarkb | we use a fip on the mirror node fwiw | 18:47 |
| dulek | fungi: Ha, I actually don't have it locally. But I can try to connect from another machine or maybe mobile tethering will have it. | 18:48 |
| fungi | dulek: okay, i'll add your key in that case | 18:48 |
| dulek | fungi: Great! | 18:48 |
| fungi | dulek: ssh root@2607:ff68:100:54:f816:3eff:fe87:3571 | 18:49 |
| corvus | clarkb: replied on 649152 -- i don't think those are errors, but doing the check first is a fine suggestion | 18:49 |
| corvus | clarkb: can you look at my reply and let me know how you would like to proceed? | 18:49 |
| fungi | dulek: if you can't manage to get ipv6 connectivity sorted out, let me know and i can always readd the autohold and we can roll the dice again | 18:49 |
| fungi | clarkb: i'm not sure if we have any available floating ips, though we only had at most two there | 18:50 |
| dulek | fungi: Okay, it's quite late where I work from, so I hope you can hold this VM until next morning? | 18:50 |
| clarkb | corvus: if we only intend for that ot run in a pipeline with a branch attribute may we shouldn't check at all? or do the check first? | 18:50 |
| fungi | dulek: sure, not a problem | 18:50 |
| corvus | clarkb: (i will be pushing up another ps to fix the pass-to-parent bug) | 18:50 |
| dulek | fungi: Great, will message you once I'm finished with it tomorrow. | 18:51 |
| fungi | thanks! | 18:51 |
| corvus | clarkb: yeah, my thinking is that both of those playbooks will fail if used in the wrong place as written. | 18:51 |
| clarkb | corvus: when scanning code like that it seems buggy to check if a var is defined in one place but not another and either not checking at all or checking once with fast fail avoids that | 18:51 |
| corvus | clarkb: yeah, it's just that the root-marker thing gets copied around a bit, so i wanted to keep those checks with it | 18:51 |
| corvus | it should really be a shared role, and it is -- in openstack-zuul-jobs | 18:52 |
| *** wolverineav has quit IRC | 18:52 | |
| corvus | i could probably copy that role into opendev/base-jobs | 18:52 |
| corvus | then the check would ride with the root-marker creation, but be out of those playbooks | 18:52 |
| clarkb | that would work too | 18:53 |
| *** wolverineav has joined #openstack-infra | 18:53 | |
| corvus | okay, let's take that as a given -- then, should we put an explicit check at the start of those playbooks to produce a nice error if they're used in the wrong place, or just leave them as-is (such that they'll crash)? | 18:53 |
| clarkb | a nice error would probably help people when bootstrapping if they've done something funny | 18:55 |
| clarkb | I'm +1 on a nice error | 18:55 |
| corvus | okay, i'll rework based on that | 18:55 |
| *** tbachman has quit IRC | 18:55 | |
| *** wolverineav has quit IRC | 18:58 | |
| *** wolverineav has joined #openstack-infra | 19:03 | |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: Rework docs jobs to be consumable by tenants https://review.openstack.org/649152 | 19:03 |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: Add write-root-marker role https://review.openstack.org/649162 | 19:03 |
| corvus | clarkb: ^ | 19:04 |
| *** wolverineav has quit IRC | 19:07 | |
| clarkb | corvus: +2 on both, thanks | 19:09 |
| *** eharney has quit IRC | 19:09 | |
| clarkb | I'm going to pop out for lunch things now. Back in a bit | 19:11 |
| *** tbachman has joined #openstack-infra | 19:18 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Make local dib file cleanup method a static method https://review.openstack.org/649166 | 19:27 |
| *** e0ne has joined #openstack-infra | 19:27 | |
| *** rascasoft has quit IRC | 19:32 | |
| openstackgerrit | Merged opendev/base-jobs master: Add write-root-marker role https://review.openstack.org/649162 | 19:32 |
| openstackgerrit | Merged opendev/base-jobs master: Rework docs jobs to be consumable by tenants https://review.openstack.org/649152 | 19:32 |
| *** rascasoft has joined #openstack-infra | 19:34 | |
| *** spsurya has quit IRC | 19:36 | |
| *** rlandy is now known as rlandy|brb | 19:36 | |
| *** jamesmcarthur has joined #openstack-infra | 19:39 | |
| *** wolverineav has joined #openstack-infra | 19:41 | |
| *** wolverineav has quit IRC | 19:43 | |
| *** wolverineav has joined #openstack-infra | 19:43 | |
| *** jcoufal has quit IRC | 19:45 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Fix for image build leaks https://review.openstack.org/647599 | 19:53 |
| *** rlandy|brb is now known as rlandy | 19:54 | |
| clarkb | corvus: fungi: should I worry about dns before sending test email through the upgraded test lists server? | 20:06 |
| clarkb | or is this a case where we'll probably be fine for a small number of emails? | 20:06 |
| corvus | clarkb: is it going to a test list with a small number of subscribers? | 20:06 |
| clarkb | corvus: I figured I'd use the infra list? | 20:07 |
| *** ijw has joined #openstack-infra | 20:07 | |
| clarkb | I guess I can manually create a test list there and add some members like myself you and fungi? | 20:07 |
| corvus | clarkb: ack. you might get a large number of bounces if you use the infra list. | 20:08 |
| corvus | might want to make a new one :) | 20:08 |
| corvus | that'll exercise more stuff anyway :) | 20:08 |
| clarkb | ok I'll start with a new one then | 20:08 |
| dmsimard | I'm working on the ssh host key thing and I'm not sure how to actually land it... the role is in zuul-jobs, if I add a new required parameter, the job from zuul-jobs will fail -- so I'd need to update the job from zuul-jobs but for it to work, it needs to depend on the role :( | 20:11 |
| dmsimard | er, let me try that again | 20:12 |
| corvus | (i'm not sure what "the ssh host key thing" is) | 20:12 |
| dmsimard | the role is in zuul-jobs, if I add a new required parameter, the job from openstack-zuul-jobs will fail -- so I'd need to update the job from openstack-zuul-jobs but for it to work, it needs to depend on the role | 20:12 |
| clarkb | corvus: something like `MAILMAN_SITE_DIR=/srv/mailman/openstack newlist` follow the prompts for upgrate-test@lists.openstack.org ? | 20:12 |
| *** verdurin has quit IRC | 20:12 | |
| dmsimard | corvus: apologies, we found out that we need ssh host keys for the upload-git-mirror role, otherwise we fail on unknown host key | 20:12 |
| clarkb | dmsimard: it is already failing right? | 20:13 |
| clarkb | dmsimard: so its ok if it fails until you add the new required car? | 20:13 |
| dmsimard | clarkb: the ara-upload-git-mirror job is already failing, yes -- but not the job that actually tests the role (zuul-jobs-upload-git-mirror) | 20:13 |
| dmsimard | that's because for the sake of testing, there was an actual keyscan in the integration test job | 20:14 |
| clarkb | in that case you should be able to update the test job so that it passes an argument value with arbitrary data (probably has to be a valid key to make ssh happy) | 20:16 |
| clarkb | then update the role, then clean up the test? | 20:16 |
| *** verdurin has joined #openstack-infra | 20:16 | |
| corvus | clarkb: yes i think so (i just checked the ruby source and i think that's close enough | 20:16 |
| dmsimard | clarkb: yeah I think we can do that. I was trying to see if there was a way without extra patches | 20:16 |
| clarkb | ok list created. Starting exim and mailman services now | 20:18 |
| *** Krenair has quit IRC | 20:22 | |
| *** xek has quit IRC | 20:22 | |
| *** rh-jelabarre has quit IRC | 20:22 | |
| clarkb | corvus: 'Recipient address rejected: Temporary deferral, try again soon' I get that from exim trying to send me the new list admin email | 20:25 |
| clarkb | is that the sort of error we would expect from my mail host if they don't like the dns setup? seems like the rejection in that case would be more forceful | 20:25 |
| *** rh-jelabarre has joined #openstack-infra | 20:26 | |
| clarkb | also there are messages like 2019-04-01 20:22:30 1hB3Ru-0008Ns-As == list@clarkb-test-lists-upgrade R=localuser T=local_delivery defer (2): No such file or directory: failed to chdir to /var/list | 20:26 |
| clarkb | though maybe that was me trying to start the mailman service and not the mailman-openstack service? | 20:26 |
| corvus | clarkb: re tempfail -- that's likely greylisting due to using an ip with no reputation | 20:27 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Pass ssh host key to upload-git-mirror for integration tests https://review.openstack.org/649183 | 20:28 |
| clarkb | 166.78.27.52 is the test server ip fwiw if anyone wants ot look at it too | 20:28 |
| *** whoami-rajat has quit IRC | 20:30 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Add parameter to upload-git-mirror to specify ssh host key https://review.openstack.org/649033 | 20:31 |
| *** kgiusti has left #openstack-infra | 20:33 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Add parameter to upload-git-mirror to specify ssh host key https://review.openstack.org/649033 | 20:35 |
| dmsimard | corvus: ^ I had forgotten the job docs after your +2 | 20:35 |
| *** jento has left #openstack-infra | 20:36 | |
| corvus | done | 20:37 |
| dmsimard | thanks <£ | 20:39 |
| dmsimard | er, <3 | 20:39 |
| clarkb | corvus: do you expect that my mx server (fastmail) will eventually let that through if I wait for the retry period or are we going to need to make external changes? | 20:39 |
| corvus | clarkb: there's a good chance that it will | 20:41 |
| corvus | clarkb: do you need it? | 20:41 |
| clarkb | 2019-04-01 20:39:31 1hB3TM-0008QJ-50 Completed | 20:41 |
| clarkb | corvus: no I just wanted to make sure things were happy before adding you and funig as testers :) | 20:41 |
| clarkb | corvus: fungi: I'll add you two as members now then work on a test email | 20:41 |
| *** trown is now known as trown|outtypewww | 20:42 | |
| clarkb | ok I think I managed to send email | 20:53 |
| clarkb | but not sure it sent to you two even after the add_members command | 20:53 |
| clarkb | did either of you get my test message? | 20:58 |
| clarkb | list_members indicates I didn't do that wrong. Though maybe even after manually adding you, you need to go through the process of accepting the subcription? I didn't send the welcome message beacuse responding to it won't work | 20:59 |
| corvus | clarkb: when did you send the message? | 20:59 |
| clarkb | corvus: about 20:50UTC | 20:59 |
| fungi | okay, i'm back from cooking/eating/chores. will look for test e-mail now | 21:00 |
| corvus | clarkb: i think your message is in moderation | 21:01 |
| clarkb | oh | 21:01 |
| clarkb | ya I guess list_members doesn't show me as a member. I sort of assumed list admins were magical | 21:02 |
| corvus | clarkb: there are 2 messages in exim's queue, you can see them in /var/spool/exim4/input/ | 21:02 |
| corvus | one of them says: 076 Subject: =?utf-8?q?Your_message_to_Upgrate-test_awaits_moderator_approval?= | 21:02 |
| corvus | (in exim's queue, data and headers are in separate files named similarly but with '-D' and '-H' suffixes) | 21:03 |
| *** e0ne has quit IRC | 21:03 | |
| clarkb | ok so if I login via the web interface and moderate myself through in theory you two get the email | 21:03 |
| * clarkb tries this | 21:03 | |
| corvus | ya | 21:03 |
| *** eglute has joined #openstack-infra | 21:04 | |
| clarkb | ok thats done | 21:05 |
| clarkb | and the moderation approval email went through after my greylist wait period | 21:05 |
| corvus | fungi and my mta's both said OK | 21:06 |
| clarkb | yup see that in mainlog now | 21:06 |
| fungi | received! | 21:06 |
| corvus | i find it very interesting that the body is base64 encoded | 21:06 |
| fungi | X-Spam-Status: No, score=1.3 required=5.0 tests=BAYES_00,BODY_URI_ONLY,FSL_HELO_NON_FQDN_1,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,MISSING_HEADERS,PYZOR_CHECK,RDNS_NONE,TVD_SPACE_RATIO | 21:06 |
| fungi | (for the record) | 21:07 |
| clarkb | I'm adding myself to the list now for further testing | 21:07 |
| fungi | oh, yeah, lookie there it's base64 indeed | 21:07 |
| corvus | clarkb: was your message body originally b64? | 21:08 |
| clarkb | corvus: no I wrote it as you suggested in your paste | 21:08 |
| fungi | Content-Transfer-Encoding: base64 | 21:08 |
| clarkb | http://paste.openstack.org/show/748274/ | 21:08 |
| corvus | interesting, i guess mailman did that | 21:08 |
| clarkb | that is gonna break dmarc if it does it globally right? | 21:08 |
| * clarkb looks at admin interface | 21:09 | |
| fungi | i have a feeling it's concerned about the lack of an actual message body | 21:09 |
| clarkb | oh maybe I can try another with a message after the subject | 21:09 |
| clarkb | but first checking if that is a toggle in the admin interface | 21:09 |
| corvus | oh, also... i think it shouldn't have a blank line before the subject, but rather, after it | 21:10 |
| fungi | in test messages of that nature i usually at least include from, to, date and message-id in addition to subject, and i also add a message body | 21:10 |
| corvus | sorry | 21:10 |
| fungi | and yeah, blank line goes between the headers and the body, not immediately after the data command | 21:10 |
| clarkb | corvus: ya I removed that blank line entirely | 21:11 |
| clarkb | I guess that may be part of the trouble? Should I send another with some body content? | 21:11 |
| corvus | yeah, i think so, lack thereof might be causing the b64 stuff | 21:12 |
| clarkb | ok and that blank line can be any text right? | 21:12 |
| clarkb | it is the actual message body | 21:12 |
| fungi | our adventures in dkim signature validation have taught us mailman does all sorts of interesting message mangling we didn't previously know about | 21:12 |
| *** owalsh_ has joined #openstack-infra | 21:12 | |
| fungi | yeah, after the blank line whatever you enter is the message body | 21:12 |
| corvus | http://paste.openstack.org/show/748690/ | 21:12 |
| fungi | up until a line with just a . on it | 21:12 |
| clarkb | ok sent | 21:14 |
| *** jamesmcarthur has quit IRC | 21:14 | |
| clarkb | raw message still seems to be base64 | 21:15 |
| clarkb | weird | 21:15 |
| *** owalsh has quit IRC | 21:15 | |
| fungi | i don't seem to have received it yet | 21:15 |
| clarkb | oh did you get moderated again? I added myself as a member /me checks | 21:16 |
| clarkb | ya I got moderated again, sorry | 21:16 |
| fungi | now received | 21:16 |
| *** owalsh has joined #openstack-infra | 21:17 | |
| fungi | https://grokbase.com/t/python/mailman-users/089bebmeek/base64-encoding | 21:17 |
| *** owalsh_ has quit IRC | 21:17 | |
| fungi | that's a really old discussion | 21:17 |
| fungi | over a decade now | 21:17 |
| *** pcaruana has quit IRC | 21:17 | |
| clarkb | I'm connecting with telnet | 21:18 |
| fungi | it may be inferring | 21:18 |
| clarkb | perhaps a control character is being seen as utf8 and getting converted? | 21:18 |
| fungi | you might set Content-Transfer-Encoding: 8bit in your headers | 21:18 |
| fungi | or maybe a content-type would solve it | 21:19 |
| fungi | Content-Type: text/plain; charset="UTF-8" | 21:19 |
| fungi | something along those lines | 21:20 |
| clarkb | and use nc | 21:20 |
| clarkb | I need to pop out for a quick break. I'll try nc + content type and encoding headers next | 21:20 |
| fungi | but yeah, i expect this is just mailman trying to be ultra safe and assume the worst because the test message doesn't provide enough context about what it should be able to assume | 21:21 |
| *** jamesmcarthur has joined #openstack-infra | 21:27 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: letsencrypt support https://review.openstack.org/636759 | 21:27 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Add graphite01.opendev.org letsencrypt configuration https://review.openstack.org/648288 | 21:27 |
| *** wolverineav has quit IRC | 21:27 | |
| ianw | fungi: ^ getting really close ... i haven't yet responded to review comments but yesterday i issued a certificate successfully using the cnames and signing domain | 21:28 |
| fungi | i saw you mention it in scrollback, excellent progress! | 21:29 |
| fungi | and i'm not sure if you saw, but i did obtain a traditional one-year cert for review.opendev.org in case we need it in a couple weeks, so don't worry that it has to be in production for the opendev move maintenance | 21:30 |
| clarkb | fungi: corvus I set content type and encoding and it still converted to base64 | 21:30 |
| fungi | huh | 21:30 |
| clarkb | I wonder if there are mailman logs we can use to figure out its decision making there | 21:30 |
| fungi | what if i try sending a message through that test ml with a traditional mua? | 21:30 |
| fungi | any objections? | 21:31 |
| clarkb | fungi: not from me | 21:31 |
| *** wolverineav has joined #openstack-infra | 21:31 | |
| clarkb | and let me know if I need to push the thing through moderation (it was made at me for implicit destination) | 21:31 |
| fungi | that's because you didn't include a to or cc header | 21:31 |
| ianw | fungi: yep, cool. this really does just get the certificate on the host, so the question of how to integrate that into something more complex like review.o.o is open ... it seems some puppet hackery will be required | 21:32 |
| clarkb | fungi: note the typo in the list name too | 21:32 |
| clarkb | it is upgrate-test not upgrade-test | 21:32 |
| clarkb | beacuse I can't type | 21:33 |
| fungi | yeah, i'm going to reply to one of your tests anyway | 21:33 |
| ianw | zigo: i should be able to hack in a buster test build this afternoon ... i would like to know that it sort of works before we put it in production, because constant failures of builds tends to have a negative affect on the builders and annoying cleanups | 21:34 |
| fungi | clarkb: 2001:4800:7811:513:be76:4eff:fe04:81b9 is the server, yeah? | 21:34 |
| clarkb | fungi: ya that looks right | 21:35 |
| clarkb | or 166.78.27.52 | 21:35 |
| *** samueldmq has joined #openstack-infra | 21:36 | |
| *** wolverineav has quit IRC | 21:36 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: web: add tenant and project scoped, JWT-protected actions https://review.openstack.org/576907 | 21:37 |
| fungi | hrm, i'm having trouble convincing my exim to look at /etc/hosts | 21:38 |
| fungi | i wonder if it's going direct to dns lookups | 21:38 |
| fungi | okay, i'll do a hybrid solution | 21:39 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: [WIP] Allow operator to generate auth tokens through the CLI https://review.openstack.org/636197 | 21:39 |
| clarkb | have mua write to file then dump that to nc ? | 21:39 |
| fungi | close | 21:39 |
| fungi | my mua already saves a copy, so yeah basically that | 21:40 |
| clarkb | ianw: exciting re letsencrypt | 21:41 |
| clarkb | ianw: should I wait on reviews considering review comments need addressing? | 21:41 |
| fungi | that worked, and it still b64 encoded the body | 21:42 |
| fungi | that was with Content-Type: text/plain; charset=us-ascii supplied (as mutt sets automatically for me) | 21:43 |
| *** ccamacho has quit IRC | 21:43 | |
| ianw | clarkb: yep, i will fix those up in a little. i've just been hacking in skips for the times when we have nothing to create/renew; now I have the valid cert i have just been confirming the scripts handle that case | 21:43 |
| fungi | included headers: date, from, to, subject, message-id, mime-version, content-type, content-disposition, user-agent | 21:43 |
| *** wolverineav has joined #openstack-infra | 21:44 | |
| *** slaweq has quit IRC | 21:45 | |
| *** mrhillsman is now known as mrhillsman_afk | 21:46 | |
| cmurphy | clarkb: remind me the etherpad for tracking trusty vs xenial? so i can stick to proposing puppet upgrades only for xenial | 21:47 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: Allow operator to generate auth tokens through the CLI https://review.openstack.org/636197 | 21:47 |
| clarkb | cmurphy: https://etherpad.openstack.org/p/201808-infra-server-upgrades-and-cleanup | 21:48 |
| clarkb | and thanks | 21:48 |
| cmurphy | ty | 21:48 |
| clarkb | fungi: corvus: are we even sure that it is mailman doing it? It could be exim right? | 21:49 |
| clarkb | fwiw my moderator approval required notices are also base64 | 21:50 |
| fungi | seems unlikely for exim to change the encoding of the message body | 21:51 |
| fungi | if it helps, this isn't necessarily a regression | 21:52 |
| *** eharney has joined #openstack-infra | 21:52 | |
| fungi | i just saw a message arrive through the starlingx-discuss ml with its message body base64-encoded too (no clue if it arrived that way, but i think we have dumps we can check in production) | 21:52 |
| fungi | nevermind, i suppose we only did that on openstack-discuss | 21:53 |
| clarkb | fungi: I checked an openstack-discuss message from earlier today and it was not base64 encoded | 21:53 |
| clarkb | but that is sample size of one | 21:53 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: Zuul CLI: allow access via REST https://review.openstack.org/636315 | 21:53 |
| fungi | yeah, i hit a few and didn't see any but i'll dig deeper | 21:53 |
| corvus | clarkb: you could shut down mailman, send a message, then we can inspect mailman's queue | 21:53 |
| clarkb | oh but my infra meeting agenda email is base64 encoded | 21:54 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Upgrade lists.katacontainers.io to puppet 4 https://review.openstack.org/647875 | 21:54 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Update review.openstack.org to puppet 4 https://review.openstack.org/647876 | 21:54 |
| clarkb | I think Content-transfer-encoding should be quoted-printable? | 21:56 |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: Add opendev-release-python job https://review.openstack.org/649199 | 21:56 |
| clarkb | fungi: ^ did you want to try another proper message with that content-transfer-encoding? I'm going to check an email you sent to prod lists to see if your mua is already doing that or if you are base64 encoded there | 21:57 |
| fungi | will do | 21:58 |
| clarkb | fungi: ya messages of yours that are not base64 encoded (like the kata/zuul feedback email) are set to quoted-printable | 21:58 |
| clarkb | and emails I send seem to be pretty consistently base64'd | 21:59 |
| clarkb | though my original copy isn't | 21:59 |
| clarkb | and I should have working dmarc just not enforcing. So you may be able to check if that makes verification sad | 22:00 |
| *** kjackal has quit IRC | 22:00 | |
| fungi | sent | 22:01 |
| clarkb | my original is Content-Type: text/plain then the copy I get back from mailman is Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 | 22:01 |
| corvus | clarkb: did you stop mailman? | 22:02 |
| fungi | i set Content-Transfer-Encoding: quoted-printable when injecting to the listserv and it came out the other end as Content-Transfer-Encoding: base64 | 22:02 |
| clarkb | corvus: I did not | 22:02 |
| corvus | that would let us confirm it's mailman doing it (it probably is, but good to be sure) | 22:03 |
| clarkb | ok I will stop it now | 22:03 |
| fungi | yeah, also it switched Content-Type: text/plain; charset=us-ascii to Content-Type: text/plain; charset="utf-8" | 22:03 |
| clarkb | hrm I don't see fungi's latest message yet | 22:04 |
| clarkb | and it isn't spam folder | 22:04 |
| * clarkb stops mailman | 22:04 | |
| clarkb | it is stopped | 22:04 |
| *** eharney has quit IRC | 22:04 | |
| fungi | we also have the option of making mailman dump verbatim copies of messages it receives for that address like we've been doing for openstack-discuss | 22:04 |
| fungi | er, making exim dump them i mean | 22:04 |
| corvus | clarkb, fungi: want to send a test message now? | 22:05 |
| clarkb | fungi: if you still have yours ready I say go for it otherwise I can do one manually | 22:06 |
| fungi | oh, yeah, and the reason you likely didn't see my second test is that i forgot to alter the previus message-id | 22:06 |
| fungi | i can do another, sure | 22:06 |
| fungi | and it should be in the queue now | 22:07 |
| *** jamesmcarthur has quit IRC | 22:08 | |
| fungi | adjusted the message-id this time so it won't appear to some recipients as a dupe | 22:08 |
| fungi | mutt showed me two copies and merely marked them as being likely duplicates, but it didn't hide one | 22:09 |
| clarkb | /srv/mailman/openstack/qfiles/in has a pck file that was just updated | 22:09 |
| *** iurygregory has quit IRC | 22:09 | |
| clarkb | is that mailmans queue? | 22:09 |
| corvus | it looks not b64 encoded, so it'll be some part of mailman doing it | 22:09 |
| corvus | yep | 22:09 |
| clarkb | agreed not base64 | 22:10 |
| clarkb | should I start mailman again? | 22:10 |
| fungi | may as well | 22:11 |
| clarkb | fwiw I'm not sure this is really an issue as long as dmarc/dkim are fine with it, but would be good to understand why it is happening | 22:11 |
| fungi | next is probably to compare output of `MAILMAN_SITE_DIR=/srv/mailman/openstack config_list -o- upgrate-test` with lists on the old server not exhibiting this behavior | 22:12 |
| fungi | and maybe see if the mailman processing pipeline grew any new entries | 22:12 |
| * clarkb does this | 22:13 | |
| *** eernst has joined #openstack-infra | 22:14 | |
| fungi | i wonder if it does this when appending the message footer to the body | 22:15 |
| corvus | perhaps this default changed: https://wiki.list.org/DOC/4.90%20My%20list%27s%20moderator%20interface%20and%20plain%20format%20digests%20have%20garbled%20non-ascii.%20Change%20English%20charset%20to%20utf-8. | 22:15 |
| clarkb | encode_ascii_prefixes = 2 on new list | 22:15 |
| clarkb | encode_ascii_prefixes = 0 on old | 22:15 |
| fungi | i thought that was only for subject lines | 22:15 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: letsencrypt support https://review.openstack.org/636759 | 22:16 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Add graphite01.opendev.org letsencrypt configuration https://review.openstack.org/648288 | 22:16 |
| clarkb | that is the only config option that looks related to me. You can see the config files in /root/clarkb-scratch | 22:17 |
| fungi | but yeah, the utf8 character set option sounds like a possibility | 22:17 |
| corvus | hrm... add_language('en', _('English (USA)'), 'us-ascii', 'ltr') | 22:17 |
| corvus | that's in /usr/lib/mailman/Mailman/Defaults.py | 22:17 |
| corvus | so i guess it didn't change | 22:17 |
| fungi | clarkb: want to try removing the message footer and seeing if that's what's causing it? | 22:17 |
| clarkb | corvus: your wiki page says if that is set then it base64 encodes | 22:18 |
| fungi | clarkb: if it's set to utf-8 instead of us-ascii | 22:18 |
| clarkb | oh right | 22:18 |
| *** eernst has quit IRC | 22:18 | |
| fungi | for ph33r of mtas which are not 8-bit clean | 22:18 |
| clarkb | starting mailman now then will remove the footer via admin interface | 22:19 |
| clarkb | hrm do we haev to set that via command line? | 22:20 |
| clarkb | oh found it under non digest options | 22:20 |
| *** mriedem has quit IRC | 22:21 | |
| clarkb | removed | 22:21 |
| fungi | okay, shall i send another? | 22:22 |
| clarkb | sure | 22:22 |
| fungi | it still switched it to utf-8 and base64 | 22:24 |
| clarkb | yup | 22:24 |
| fungi | i feel like corvus is onto something but that the setting we think should prevent it is being ignored/overridden | 22:25 |
| clarkb | fwiw http://dkim.org/specs/rfc4871-dkimbase.html#hashing implies to me that we would break dkim verification if we base64 encode globally | 22:28 |
| corvus | wow | 22:30 |
| clarkb | fungi: corvus we could set mm_cfg.py back to ascii if we suspect something is overriding the default? | 22:30 |
| corvus | clarkb: the file i linked is included in mm_cfgpy | 22:31 |
| corvus | so it's effectively already there | 22:31 |
| clarkb | ah | 22:31 |
| *** betherly has joined #openstack-infra | 22:34 | |
| *** nikrin is now known as knikolla | 22:34 | |
| fungi | yeah, i can't see where it's happening, but messages it's receiving with Content-Type: text/plain; charset=us-ascii are getting switched to Content-Type: text/plain; charset=us-ascii | 22:35 |
| corvus | >>> mm_cfg.LC_DESCRIPTIONS['en'] | 22:35 |
| corvus | ('English (USA)', 'utf-8', 'ltr') | 22:35 |
| corvus | huh | 22:35 |
| corvus | clarkb: i think your theory warrants further investigation :) | 22:35 |
| fungi | er, switched to Content-Type: text/plain; charset="utf-8" | 22:35 |
| clarkb | corvus: oh interesting. I was just verifying your assertion about loading the defaults and we do import * from Defaults | 22:36 |
| corvus | oh | 22:38 |
| corvus | def add_language(code, description, charset, direction='ltr'): | 22:38 |
| corvus | LC_DESCRIPTIONS[code] = (description, 'utf-8', direction) | 22:38 |
| corvus | apparently *every* language is utf-8 now | 22:38 |
| clarkb | heh | 22:39 |
| clarkb | magic | 22:39 |
| corvus | that same code is on the prod server though | 22:39 |
| fungi | all languages are now taco bell | 22:39 |
| *** betherly has quit IRC | 22:39 | |
| fungi | i wonder if this is being set in debian packaging | 22:39 |
| fungi | because of the rush to make all things utf-8 friendly | 22:39 |
| *** rcernin has joined #openstack-infra | 22:40 | |
| clarkb | corvus: if that was from ap ackage udpate on the prod server would existing lists have kept theri ascii value in a db somewhere? | 22:40 |
| corvus | so if the config setting is the same, i guess we're looking at a mailman behavioral change... | 22:40 |
| clarkb | and only new lists are affected by the charset override? | 22:40 |
| corvus | oh, hrm | 22:40 |
| corvus | i don't know whether or not anything related to this is stored in the pickle | 22:41 |
| clarkb | strings doesn't seem to show it as a string value at least | 22:42 |
| clarkb | do we want to edit /etc/mm_cfg.py to set it back to ascii then restart mailman-openstack and at least confirm that this "fixes" it? | 22:43 |
| corvus | that may give us more information to help narrow down what changed. but if it did "fix" it, i would be reluctant to do that in production | 22:44 |
| clarkb | ya not necessarily a good long term option but helps potentially give us a better understanding of what it is doing | 22:45 |
| clarkb | Howd does this look: would be added to end of /etc/mm_cfg.py LC_DESCRIPTIONS['en'] = ('English (USA)', 'us-ascii', 'ltr') | 22:47 |
| corvus | i think that would do it | 22:47 |
| clarkb | ok adding now and will restart mailman-openstack | 22:48 |
| corvus | i skimmed the release notes for upstream mailman and don't see any relevant changes | 22:48 |
| clarkb | ok that is done | 22:49 |
| clarkb | fungi: want to send another test? | 22:49 |
| clarkb | hrm maybe we've lost fungi. I can do another local test | 22:52 |
| corvus | drat, i was going to use the delay to see if you would review https://review.openstack.org/649173 and https://review.openstack.org/649199 :) | 22:52 |
| clarkb | ok that does fix it | 22:54 |
| clarkb | "fix" | 22:54 |
| *** rh-jelabarre has quit IRC | 22:55 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: letsencrypt support https://review.openstack.org/636759 | 22:55 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Add graphite01.opendev.org letsencrypt configuration https://review.openstack.org/648288 | 22:55 |
| clarkb | corvus: I'll review those now that we have an "answer" to the other thing | 22:55 |
| clarkb | let me write down what we've learned on my etherpad though | 22:56 |
| *** tkajinam has joined #openstack-infra | 22:56 | |
| fungi | sorry, back, had to step away for a sec | 22:57 |
| corvus | clarkb, fungi: if we decided that we didn't want to try to dig deeper into what part of debian or mailman changed to cause the new behavior, and instead just gave up and enabled the full range of dmarc mitigation, problematic though it is, i would not blame us. | 22:57 |
| clarkb | ok notes at https://etherpad.openstack.org/p/lists.o.o-trusty-to-xenial list item 5.2.5 if you want to review | 22:57 |
| clarkb | corvus: ya it certianly seems like we are being crowded into a corner by a large number of items | 22:58 |
| clarkb | you address item a and then run into b and rinse and repeat | 22:58 |
| fungi | not the least of which (after dkim itself) is mailman being disinterested in avoiding mangling messages because they have rewriting solutions to recommend for folks worried about signature invalidation | 22:59 |
| clarkb | catching up on lists upgrade status is already part of the plan for tomorrows meeting so maybe we should all sleep on it given what we have learned and talk about it tomorrow? | 22:59 |
| corvus | i confess i'm surprised; i would have thought patching a few holes would be sustainable, but for a package which is as famously slow to move as mailman is, they've managed to eliminate any alternatives which make coexistence with dmarc palatable at a shockingly rapid rate. | 22:59 |
| clarkb | we may also want to double check that dmarc/dkim verification fails on the base64 encoded emails? | 23:04 |
| clarkb | fungi: ^ iirc you had a verification setup that was easy to check on? my email today about meeting agenda on the infra list is one that you can check I think | 23:04 |
| * clarkb looks again | 23:05 | |
| clarkb | oh dmarc is none I'm only doing dkim and it did fail ebcause message was altered btu that might be the footer? | 23:06 |
| fungi | i already deleted my copy, i'm afraid | 23:06 |
| clarkb | ya the footer was added too so those are a bad example | 23:07 |
| clarkb | and discuss doesn't seem to do it? | 23:07 |
| fungi | not that i've been able to find examples of, no | 23:08 |
| clarkb | on -discuss were there any other settings we set to avoid modifying the message body or was it just the footer contents? | 23:08 |
| clarkb | because ya -discuss isn't doing this but Defaults.py there is set this way | 23:09 |
| clarkb | I suppose the older codebase could hardcode en to ascii | 23:09 |
| fungi | that's all that comes to mind | 23:09 |
| *** aaronsheffield has quit IRC | 23:11 | |
| clarkb | I've confirmed that the mailman source in the debian package does not have the charset hardcoded to utf8 | 23:14 |
| clarkb | but I don't see where debian is patching that yet | 23:14 |
| clarkb | hrm it is a .py.in file though so maybe mailman itself is doing it via some build flag | 23:18 |
| clarkb | also could be an ubuntu difference /me checks | 23:20 |
| clarkb | https://git.launchpad.net/ubuntu/+source/mailman/tree/debian/patches/91_utf8.patch?h=applied/ubuntu/xenial-updates#n41 yup ubuntu specific | 23:22 |
| clarkb | and seems to have been applied back to trusty | 23:24 |
| clarkb | could be that the change in behavior between trusty and xenial is that mailman started to base64 when utf8'd | 23:24 |
| clarkb | I get to use bzr clone :) | 23:27 |
| *** tosky has quit IRC | 23:28 | |
| clarkb | if anyone is wondering this is not fast | 23:30 |
| *** hwoarang has quit IRC | 23:35 | |
| *** hwoarang has joined #openstack-infra | 23:37 | |
| *** wolverineav has quit IRC | 23:39 | |
| *** wolverineav has joined #openstack-infra | 23:39 | |
| *** wolverineav has quit IRC | 23:44 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: letsencrypt support https://review.openstack.org/636759 | 23:44 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Add graphite01.opendev.org letsencrypt configuration https://review.openstack.org/648288 | 23:44 |
| clarkb | https://bugs.python.org/issue12552 hrm python has done this for forever | 23:50 |
| clarkb | so how is it that -discuss has 7bit transfer encoded messages on content type charset=utf8 | 23:51 |
| *** openstackgerrit has quit IRC | 23:56 | |
| *** timburke has quit IRC | 23:58 | |
| *** antonym has quit IRC | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!