| opendevreview | Merged openstack/kayobe master: Revert "CI: drop jobs requiring kolla-ansible" https://review.opendev.org/c/openstack/kayobe/+/830350 | 00:31 |
|---|---|---|
| opendevreview | James Kirsch proposed openstack/kolla-ansible master: Add support for LetsEncrypt-managed certs https://review.opendev.org/c/openstack/kolla-ansible/+/741340 | 02:02 |
| opendevreview | James Kirsch proposed openstack/kolla-ansible master: Add support for LetsEncrypt-managed certs https://review.opendev.org/c/openstack/kolla-ansible/+/741340 | 04:26 |
| opendevreview | wangxiyuan proposed openstack/kolla-ansible master: [WIP]Add openEuler Distro support https://review.opendev.org/c/openstack/kolla-ansible/+/830115 | 07:30 |
| opendevreview | Pierre Riteau proposed openstack/kayobe master: CI: test fact caching https://review.opendev.org/c/openstack/kayobe/+/808218 | 08:30 |
| frickler | yoctozepto: meh, gerrit should really give a warning when submitting a review and there has been another review while looking at the change | 08:36 |
| yoctozepto | frickler: yup | 08:37 |
| mnasiadka | morning | 09:07 |
| opendevreview | Mark Goddard proposed openstack/kolla-ansible master: [WIP] [CI] Use Tenks in Ironic job https://review.opendev.org/c/openstack/kolla-ansible/+/644271 | 09:24 |
| hrw | yo | 09:26 |
| opendevreview | Mark Goddard proposed openstack/kayobe master: DNM: test TLS https://review.opendev.org/c/openstack/kayobe/+/830566 | 09:34 |
| opendevreview | Piotr Parczewski proposed openstack/kolla-ansible master: Fix hard coded OIDC response type https://review.opendev.org/c/openstack/kolla-ansible/+/830569 | 09:57 |
| opendevreview | Radosław Piliszek proposed openstack/kolla-ansible master: [WIP] [CI] Use Tenks in Ironic job https://review.opendev.org/c/openstack/kolla-ansible/+/644271 | 10:10 |
| mgoddard | yoctozepto, mnasiadka, hrw, frickler: I wrote some thoughts on the letsencrypt patch: https://review.opendev.org/c/openstack/kolla-ansible/+/741340 | 10:18 |
| mgoddard | if any of you are able to get up to speed on it, it would be nice to discuss in today's meeting | 10:19 |
| frickler | mgoddard: ack, I have some pretty devastating opinion on this, but happy to discuss | 10:26 |
| mgoddard | frickler: tl;dr? | 10:33 |
| frickler | mgoddard: HTTP-01 is useless, I need DNS-01 for internal endpoints anyway | 10:40 |
| mgoddard | well, correct | 10:41 |
| mgoddard | but you could have an internal CA for internal endpoints | 10:42 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/xena: CI: Bump Ceph to Pacific https://review.opendev.org/c/openstack/kolla-ansible/+/828757 | 10:42 |
| frickler | but why split it up and double the work? I also have most deployments not being public, so "public" endpoints also need DNS-01 for those. and I want wildcard certs for RGW. | 10:44 |
| mgoddard | frickler: so use DNS-01. It's not always an option though | 10:53 |
| opendevreview | Merged openstack/kolla-ansible stable/xena: [CI] Check fluentd errors https://review.opendev.org/c/openstack/kolla-ansible/+/828654 | 11:06 |
| opendevreview | Merged openstack/kolla-ansible stable/wallaby: [CI] Check fluentd errors https://review.opendev.org/c/openstack/kolla-ansible/+/828655 | 11:13 |
| sri_ | hi team, When I enable prometheus in xena release I am running into this error " {{ groups['prometheus'][0] }}]: FAILED! => {"msg": "The field 'delegate_to' has an invalid value, which includes an undefined variable. The error was: list object has no element" , also I am only seeing this error on multinode deployment . | 11:25 |
| sri_ | is there any known bug or am I doing something wrong ! | 11:28 |
| opendevreview | Merged openstack/kolla-ansible master: Remove classic queue mirroring for internal RabbitMQ https://review.opendev.org/c/openstack/kolla-ansible/+/824994 | 11:43 |
| priteau | sri_: you need hosts to be added to your monitoring group | 11:44 |
| sri_ | priteau: oh, my bad. Thanks Pierre :) | 11:46 |
| opendevreview | Juan Pablo Suazo proposed openstack/kolla-ansible master: Adds services to log_rotate. https://review.opendev.org/c/openstack/kolla-ansible/+/830433 | 12:53 |
| opendevreview | Merged openstack/kayobe master: Sync enable flag defaults with kolla ansible https://review.opendev.org/c/openstack/kayobe/+/829114 | 13:31 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: docs: standard PTG topics list https://review.opendev.org/c/openstack/kolla/+/830613 | 13:41 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/xena: CI: Bump Ceph to Pacific https://review.opendev.org/c/openstack/kolla-ansible/+/828757 | 13:45 |
| opendevreview | Mark Goddard proposed openstack/kayobe stable/xena: Sync enable flag defaults with kolla ansible https://review.opendev.org/c/openstack/kayobe/+/830578 | 13:58 |
| opendevreview | Radosław Piliszek proposed openstack/kolla-ansible master: [WIP] [CI] Use Tenks in Ironic job https://review.opendev.org/c/openstack/kolla-ansible/+/644271 | 14:09 |
| opendevreview | Verification of a change to openstack/kayobe master failed: CI: enable libvirt TLS in TLS job https://review.opendev.org/c/openstack/kayobe/+/826739 | 14:14 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: docs: standard PTG topics list https://review.opendev.org/c/openstack/kolla/+/830613 | 14:44 |
| opendevreview | Merged openstack/kayobe-config-dev master: libvirt: Don't require Virtualisation Technology (VT) https://review.opendev.org/c/openstack/kayobe-config-dev/+/829225 | 14:53 |
| mnasiadka | mgoddard mnasiadka hrw egonzalez yoctozepto rafaelweingartne cosmicsound osmanlicilegi bbezak parallax Fl1nt frickler adrian-a - meeting in 6 minutes | 14:54 |
| mnasiadka | #startmeeting Kolla | 15:00 |
| opendevmeet | Meeting started Wed Feb 23 15:00:53 2022 UTC and is due to finish in 60 minutes. The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| opendevmeet | The meeting name has been set to 'kolla' | 15:00 |
| mnasiadka | #topic rollcall | 15:01 |
| yoctozepto | o/ | 15:01 |
| mnasiadka | o/ | 15:01 |
| ohorecny2 | \o | 15:02 |
| frickler | o/ | 15:02 |
| mnasiadka | #topic agenda | 15:04 |
| mnasiadka | * Announcements | 15:04 |
| mnasiadka | * Review action items from the last meeting | 15:04 |
| mnasiadka | * CI status | 15:04 |
| mnasiadka | * Release tasks | 15:04 |
| yoctozepto | mnasiadka gogo | 15:04 |
| mnasiadka | * Current cycle planning | 15:04 |
| mnasiadka | * Additional agenda (from whiteboard) | 15:04 |
| mnasiadka | * Open discussion | 15:04 |
| yoctozepto | :-) | 15:04 |
| mnasiadka | #topic Announcements | 15:04 |
| mnasiadka | I booked the same PTG slots as last time - Mon-Wed (Wed for Kayobe) - 13-17UTC (13-15 UTC on Wed) | 15:05 |
| mnasiadka | Created etherpad | 15:05 |
| mnasiadka | https://etherpad.opendev.org/p/kolla-zed-ptg | 15:05 |
| mnasiadka | #url https://etherpad.opendev.org/p/kolla-zed-ptg | 15:05 |
| mnasiadka | Please put your topic proposals in there | 15:05 |
| mgoddard | \o | 15:06 |
| yoctozepto | (psst, it's #link) | 15:06 |
| mnasiadka | ah | 15:06 |
| mnasiadka | #link https://etherpad.opendev.org/p/kolla-zed-ptg | 15:06 |
| mnasiadka | thanks yoctozepto | 15:06 |
| yoctozepto | yw mnasiadka | 15:06 |
| mnasiadka | #topic Review action items from the last meeting | 15:06 |
| mnasiadka | mnasiadka post a patch for docs - standard topics that should be discussed over PTG and then revisited in mid-cycle | 15:06 |
| mnasiadka | mnasiadka to triage security bugs and update them with resolution plan (if needed) | 15:06 |
| mnasiadka | hrw to discuss with pynacl upstream to release binary wheel of 1.4.0 for aarch64 | 15:06 |
| mnasiadka | did first, a bit - patch posted | 15:06 |
| mnasiadka | https://review.opendev.org/c/openstack/kolla/+/830613 | 15:06 |
| mnasiadka | second to be continued | 15:07 |
| mnasiadka | #action mnasiadka to triage security bugs and update them with resolution plan (if needed) | 15:07 |
| mnasiadka | #action hrw to discuss with pynacl upstream to release binary wheel of 1.4.0 for aarch64 | 15:07 |
| mnasiadka | since hrw is not here | 15:07 |
| mnasiadka | #topic CI status | 15:07 |
| mnasiadka | How is CI? | 15:07 |
| mnasiadka | Whiteboard says Kayobe CI is RED due to ping issue? | 15:07 |
| mnasiadka | (probably outdated) | 15:08 |
| yoctozepto | k and k-a seem fine | 15:08 |
| mgoddard | kob fixed | 15:08 |
| mnasiadka | thanks mgoddard | 15:08 |
| mnasiadka | #topic Release tasks | 15:09 |
| mnasiadka | Release mgmt team has asked for Cycle highlights, I'll post up a patch and ask for reviews | 15:09 |
| mnasiadka | #action mnasiadka to post patch for cycle highlights | 15:09 |
| mnasiadka | #topic Current cycle planning | 15:10 |
| mnasiadka | mgoddard: you wanted to discuss Let's Encrypt? | 15:11 |
| mnasiadka | We can do that in the additional topics slot if you prefer | 15:11 |
| mgoddard | yes | 15:11 |
| mgoddard | either is fine | 15:11 |
| mnasiadka | Ok - just a reminder: Kolla feature freeze: Mar 21 - Mar 25 | 15:13 |
| yoctozepto | it's going to be chilly in March! | 15:13 |
| mnasiadka | So let's go with Let's Encrypt | 15:13 |
| yoctozepto | let's go and let's encrypt indeed | 15:13 |
| mgoddard | has anyone reviewed the patch recently? | 15:13 |
| yoctozepto | I did not have time to read the patch | 15:13 |
| yoctozepto | I would love a tl;dr | 15:14 |
| mgoddard | I think we need a rethink. | 15:15 |
| mgoddard | I don't think we can expose the HAProxy admin socket unauthenticated via TCP | 15:15 |
| mgoddard | openstack-ansible suggests they use separate certs for each load balancer. That would avoid the sync, and greatly simplify the design. We could also use a unix admin socket. See https://docs.openstack.org/openstack-ansible/latest/user/security/ssl-certificates.html#certbot-certificates and https://opendev.org/openstack/openstack-ansible-haproxy_server | 15:15 |
| mgoddard | we need to store the certs on disk, as well as dynamically updating HAProxy. This would be a lot easier if we only had to update the local HAproxy | 15:15 |
| mgoddard | the bootstrapping process seems clumsy, and it concerns me that a reconfigure doesn't work. A colleague suggested using certbot standalone mode to bootstrap when we don't have certificates.That could be fiddly, but either way, I'd like to see a clean, documented way to bootstrap this (that ensures we don't overwrite the LE certs with our own self-signed ones). It might involve getting | 15:15 |
| mgoddard | HAProxy running first to bootstrap LE, then running another deploy with everything else. | 15:15 |
| mgoddard | the internal API support doesn't seem that useful to me, and if we're going to iterate the design then it might be easier to remove it | 15:15 |
| mgoddard | Overall, I'd like to see a written plan for the approach, that a few people can agree on - we should have enough context at this point to agree on a design. | 15:15 |
| mgoddard | a bit long for a tl;dr, but that was my summary comment | 15:15 |
| yoctozepto | I was about to say that! | 15:15 |
| * yoctozepto reading | 15:15 | |
| mnasiadka | Ok, just to be clear - we're not going to support DNS-01? only HTTP-01 challenge? | 15:16 |
| mgoddard | correct | 15:16 |
| mnasiadka | I'm not utterly happy about that. | 15:17 |
| mgoddard | at least for now | 15:17 |
| mgoddard | I don't know what's involved in DNS-01 | 15:17 |
| mnasiadka | a DNS server that can be ,,orchestrated'' or manual TXT entries in the domain | 15:18 |
| mnasiadka | I'm just saying it might be even easier - and that's required for wildcard certificates | 15:18 |
| mnasiadka | We don't need to expose anything. | 15:18 |
| mgoddard | that's about as much as I know about DNS-01 | 15:19 |
| yoctozepto | the problem with DNS-01 and k-a is that k-a does not care about the user's DNS server | 15:19 |
| mgoddard | what I don't know is whether we could provide any form of general support for it | 15:19 |
| yoctozepto | mgoddard paraphrased me | 15:20 |
| mnasiadka | With certbot and it's semi-broken support for any normal forms of DNS-01, it might be complicated. | 15:20 |
| mgoddard | it's proving difficult enough to implement HTTP-01. If you'd like to ask James to implement DNS-01 too he might not be wild about it | 15:22 |
| yoctozepto | what is the admin socket on tcp for? | 15:23 |
| mgoddard | to update the certs dynamically | 15:24 |
| mnasiadka | So, my problem is currently, that with the merged patch to Kolla - we're limiting ourselves to certbot (which in most cases won't work for most DNS-01 providers). I'm fine with first doing HTTP-01 and then DNS-01 (if it's possible to add later). | 15:24 |
| mgoddard | this patch has been around for some time, and this is the first time I'm hearing a request for DNS-01 | 15:25 |
| frickler | couldn't the cert updates be done by a service container similar to e.g. keystone-fernet? that would need the admin socket neither via tcp nor on the host I think | 15:26 |
| mgoddard | does anyone know how many deployments would be likely to use HTTP-01 vs DNS-01? | 15:26 |
| mgoddard | frickler: it was like that in a previous iteration | 15:26 |
| mgoddard | it seems that openstack-ansible just uses a different cert for each host, and avoids syncing | 15:27 |
| mgoddard | that seems like a great simplifier to me | 15:27 |
| mgoddard | probably we should look at their implementation | 15:27 |
| mgoddard | (we == headphoneJames) | 15:27 |
| frickler | for HTTP-01 vs. DNS-01, my deployments all would use the latter, but I also consider that to be out of the scope of k-a. I just need a nice interface to rotate the certs I refreshed outside of kolla | 15:28 |
| yoctozepto | yeah, cert rotation is probably one thing to tackle | 15:28 |
| frickler | for a general survey, does it make sense to add that question to the openstack user survey? would be some time until we get results, though | 15:29 |
| mgoddard | probably too long, although this patch has been around for some time | 15:30 |
| mnasiadka | yes, but from what I understand (from headphoneJames' email) HAProxy 2.2 is rejecting multi certificate pem files in the ''hot reload'' feature? | 15:30 |
| mnasiadka | maybe frickler is right - we just need to focus on means to dynamically update certificates - who cares if a user is using certbot or not. | 15:31 |
| frickler | mnasiadka: do you have a link to that email? | 15:32 |
| mnasiadka | frickler: no, that was shared private - I can forward | 15:33 |
| frickler | ah, that explains why I didn't see it ;) | 15:34 |
| mnasiadka | https://www.mail-archive.com/haproxy@formilux.org/msg40150.html | 15:35 |
| mnasiadka | a bit related to single file with multiple certs ;-) | 15:35 |
| mnasiadka | So - is there any rough plan for that feature? | 15:37 |
| headphoneJames | Fyi, 2.2 did turn out to support dynamic reload | 15:38 |
| mgoddard | sorry, had to run - poorly child | 15:40 |
| yoctozepto | mgoddard: understandable! best wishes! | 15:41 |
| mgoddard | what do we mean by dynamic reload without certbot here though? how would new certs get placed? | 15:42 |
| mnasiadka | user-provided mechanism, for those that don't want to use certbot ;-) | 15:43 |
| mnasiadka | just a kolla-ansible command to update the certs to newly uploaded ones? | 15:43 |
| mgoddard | I suppose we could drop certs to /etc/kolla/haproxy/haproxy.pem, then provide a script to do the dynamic reload | 15:43 |
| mnasiadka | sounds good to me, that gives us some functionality we could merge this cycle? | 15:45 |
| mgoddard | potentially | 15:45 |
| mgoddard | assuming headphoneJames is on board | 15:46 |
| headphoneJames | Would we make cert bot available to kolla Ansible to generate certs? | 15:46 |
| opendevreview | Radosław Piliszek proposed openstack/kolla-ansible master: [WIP] [CI] Use Tenks in Ironic job https://review.opendev.org/c/openstack/kolla-ansible/+/644271 | 15:47 |
| mnasiadka | certbot container patch is merged already | 15:48 |
| mgoddard | yes, but that's the easy part :) | 15:48 |
| yoctozepto | yeah | 15:50 |
| mnasiadka | as I said, I'm not a certbot user - I can understand it can fit some cases - but I'd like to also have the option of not using it - and having a separate mechanism delivering the certs to haproxy and just signalling that it should reload the cert ;-) | 15:50 |
| headphoneJames | From what in reading, It sounded like the certificates would be generated during deployment instead of after container is deployed | 15:50 |
| mnasiadka | if we can have reliable automation for the certbot part - I'm all in (but maybe these should be separate patches) | 15:51 |
| mgoddard | if someone can write up how dynamic reload would work in a way that would be generally useful, that would be helpful | 15:51 |
| mgoddard | is it still using certs on the deployment host and copying those across, or does it assume some process has put them into place on the haproxy hosts? | 15:52 |
| mnasiadka | so, for dns-01 case, it would be nice if kolla-ansible would copy out the cert to nodes and update them in haproxy | 15:53 |
| headphoneJames | I'm assuming the former based on this conversation | 15:54 |
| mnasiadka | frickler: opinions? | 15:54 |
| mgoddard | the former doesn't really work with HTTP-01 | 15:54 |
| frickler | I'm not sure how the dynamic update works | 15:54 |
| mnasiadka | #link https://www.haproxy.com/blog/dynamic-ssl-certificate-storage-in-haproxy/ | 15:55 |
| mnasiadka | mgoddard: for http-01 we need to stand up a backend on each of the hosts that serve haproxy? | 15:55 |
| mgoddard | typically, yes | 15:56 |
| yoctozepto | afair, we were discussing that we need only https://www.haproxy.com/blog/hitless-reloads-with-haproxy-howto/ | 15:56 |
| frickler | I was just wondering whether hitless reload wouldn't be good enough in our case | 15:57 |
| yoctozepto | (as the mnasiadka's linked post suggests to use if one does not have many many certs) | 15:57 |
| yoctozepto | and that is what we discussed | 15:57 |
| frickler | the dynamic update seems a bit overkill | 15:57 |
| mgoddard | makes sense | 15:57 |
| yoctozepto | the issue was we did not have the possibility to reload | 15:57 |
| yoctozepto | and still do not have | 15:57 |
| yoctozepto | I mean, in k-a | 15:57 |
| yoctozepto | the reason was the file copying | 15:58 |
| yoctozepto | as the certs have to be first copied into the running container | 15:58 |
| mgoddard | true that | 15:58 |
| yoctozepto | it seems the patch has grown much beyond the original plan | 15:58 |
| mgoddard | well 1 minute to go | 15:59 |
| frickler | can't we bindmount the certs in and update them on the host? | 15:59 |
| mnasiadka | 1 minute to go, yes | 15:59 |
| mnasiadka | should we have some dedicated meeting for this? | 15:59 |
| yoctozepto | 5 sec | 15:59 |
| yoctozepto | and go | 16:00 |
| yoctozepto | dedicated meeting ++ | 16:00 |
| yoctozepto | the PTG | 16:00 |
| yoctozepto | :D | 16:00 |
| mnasiadka | probably we would like to have something merged this cycle :D | 16:00 |
| mnasiadka | ok, let's discuss about the dedicated meeting after the official meeting :D | 16:00 |
| yoctozepto | yeah, true that | 16:00 |
| yoctozepto | ++ | 16:00 |
| mnasiadka | thanks for joining, sorry for not covering all topics... | 16:00 |
| mnasiadka | #endmeeting | 16:00 |
| opendevmeet | Meeting ended Wed Feb 23 16:00:58 2022 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:00 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/kolla/2022/kolla.2022-02-23-15.00.html | 16:00 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/kolla/2022/kolla.2022-02-23-15.00.txt | 16:00 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/kolla/2022/kolla.2022-02-23-15.00.log.html | 16:00 |
| yoctozepto | thanks mnasiadka | 16:01 |
| mnasiadka | maybe a more interactive meeting like audio call beginning next week for the Let's Encrypt feature? | 16:01 |
| mgoddard | FWIW, here is the original spec: https://etherpad.opendev.org/p/kolla-ansible-letsencrypt-https | 16:02 |
| headphoneJames | Fyi I'm out until Thursday next week | 16:03 |
| opendevreview | Radosław Piliszek proposed openstack/kolla-ansible master: [WIP] [CI] Use Tenks in Ironic job https://review.opendev.org/c/openstack/kolla-ansible/+/644271 | 16:55 |
| opendevreview | Radosław Piliszek proposed openstack/kolla-ansible master: [WIP] [CI] Use Tenks in Ironic job https://review.opendev.org/c/openstack/kolla-ansible/+/644271 | 16:56 |
| opendevreview | Radosław Piliszek proposed openstack/kolla-ansible master: [WIP] [CI] Use Tenks in Ironic job https://review.opendev.org/c/openstack/kolla-ansible/+/644271 | 16:56 |
| opendevreview | Merged openstack/kayobe master: CI: enable libvirt TLS in TLS job https://review.opendev.org/c/openstack/kayobe/+/826739 | 18:00 |
| opendevreview | Juan Pablo Suazo proposed openstack/kolla-ansible master: Adds services to log_rotate. https://review.opendev.org/c/openstack/kolla-ansible/+/830433 | 20:10 |
| opendevreview | Radosław Piliszek proposed openstack/kolla-ansible master: [WIP] [CI] Use Tenks in Ironic job https://review.opendev.org/c/openstack/kolla-ansible/+/644271 | 20:18 |
| opendevreview | Juan Pablo Suazo proposed openstack/kolla-ansible master: Adds services to log_rotate. Fixes Bug 1961795. https://review.opendev.org/c/openstack/kolla-ansible/+/830433 | 22:23 |
| opendevreview | Juan Pablo Suazo proposed openstack/kolla-ansible master: Adds services to log_rotate. https://review.opendev.org/c/openstack/kolla-ansible/+/830433 | 22:24 |
| opendevreview | Pierre Riteau proposed openstack/kayobe master: ntp: Remove removal of chrony container https://review.opendev.org/c/openstack/kayobe/+/827487 | 22:36 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!