| opendevreview | Jan Gutter proposed openstack/kolla-ansible master: WIP debug kolla-ansible-ubuntu-upgrade-cephadm https://review.opendev.org/c/openstack/kolla-ansible/+/890822 | 02:19 |
|---|---|---|
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: systemd: handle running container without systemd unit https://review.opendev.org/c/openstack/kolla-ansible/+/890198 | 05:57 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: DNM: dumb-init verbose https://review.opendev.org/c/openstack/kolla/+/890876 | 06:08 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: systemd: handle running container without systemd unit https://review.opendev.org/c/openstack/kolla-ansible/+/890198 | 07:06 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: DNM: Test with docker_restart_policy default https://review.opendev.org/c/openstack/kolla-ansible/+/890658 | 07:06 |
| opendevreview | howardlee proposed openstack/kolla-ansible master: Dev mode support for cyborg https://review.opendev.org/c/openstack/kolla-ansible/+/890883 | 07:18 |
| SvenKieske | mnasiadka: you got comments - well one comment - I'm curious why the removal of docker containers should actually fail, maybe it's related to cadvisor. | 07:34 |
| mnasiadka | SvenKieske: so, currently we delete it with force=True, which sends SIGKILL if the container is up - because we don't really wait for the container to be stopped, so in some extreme cases - the container gets killed on removal - it's seen extremely well with mariadb upgrade (because you can't upgrade a database that has crashed) | 07:36 |
| mnasiadka | and yes, I think the case that removal with force=True fails is some old bug, but better to leave it there, than be sorry later | 07:36 |
| SvenKieske | and we send sigkill, because sigterm does run too long, or..? | 07:37 |
| mmalchuk | morning o/ | 07:37 |
| SvenKieske | morning o/ | 07:37 |
| mmalchuk | mnasiadka what with https://review.opendev.org/c/openstack/kolla-ansible/+/888943 ? | 07:37 |
| mnasiadka | mmalchuk: to have a baseline - mod_status is disabled by default on Rocky/CentOS? | 07:39 |
| mmalchuk | can't check, but imho yes | 07:39 |
| mmalchuk | it enabled in Ubuntu/Debian in postinstall scripts of the package | 07:40 |
| mmalchuk | Rocky/Centos afaik dont enable anything by default | 07:40 |
| mnasiadka | well, then let's make it in the same state for all distributions | 07:43 |
| mmalchuk | you mean disable mod_status? but there are many people use it for monitoring... me too! | 07:45 |
| mmalchuk | or enable mod_status in rocky/centos ? and merge fix? | 07:46 |
| mnasiadka | so enable that for RedHat clones as well | 07:46 |
| mmalchuk | ok. will do this as a followup | 07:47 |
| mmalchuk | lets merge security fix | 07:47 |
| mnasiadka | I commented on the patch | 07:49 |
| SvenKieske | I also did comment | 07:53 |
| SvenKieske | this is not only about security but about exposing user info, which is PII under gdpr, so it would be really nice to get this merged fast and finetune later. | 07:53 |
| mnasiadka | I don't like running like headless chicken and merging something + backporting that everywhere, just to fix something, that has been there for x years | 07:57 |
| mnasiadka | and then revisiting it ;) | 07:57 |
| SvenKieske | well I agree and disagree: I don't see any headless chicken, so disagree. I also think this should've been catched when it was implemented, so agreed it should never have been there for x years, does nobody do port scans these days anymore? | 07:59 |
| SvenKieske | that was how I found that rabbitmq by default listens on all interfaces in k-a; which is just bad, and the facility to actually configure this was broken for some releases. all I did was a portscan of haproxy. | 08:00 |
| SvenKieske | so maybe we should be more cautious in general when accepting new features. :/ | 08:01 |
| mmalchuk | mnasiadka but we didn't know that it was there for x years! this is security flaw not seed for x years | 08:01 |
| mnasiadka | mmalchuk: stay calm, breathe | 08:01 |
| mmalchuk | and you are not headless chicken, I've and kevko spend a lot of time to invecstigate and fix the issue | 08:02 |
| * mmalchuk calm | 08:02 | |
| mnasiadka | mmalchuk: we never fixed that, so I assume it was there for x years, I doubt Ubuntu and Debian decided just now to enable mod_status by default | 08:02 |
| mmalchuk | no! this appears in 2.4 apache last years | 08:02 |
| mnasiadka | 2.4 was released 11 years ago | 08:03 |
| mmalchuk | ok x=11 | 08:03 |
| SvenKieske | I really get mad when people tell me "it's okay, this issue has been there for x years", like do you realize a security issue that's been sitting there for x years makes it actually worse, not better? | 08:04 |
| * SvenKieske calm | 08:04 | |
| mnasiadka | so you want to tell me, that although we configure backend to an internal ip, the traffic goes to 127.0.0.1? | 08:04 |
| mmalchuk | we can check when he decide to enable by default, but this is not solve the issue. there many peoples already know about flaw an use it for enumerate attack | 08:05 |
| mmalchuk | not realy 127.0.0.1 | 08:05 |
| SvenKieske | no, at least in older releases rabbitmq listened on 0.0.0.0 which is all interfaces. | 08:05 |
| mmalchuk | there in config 'Require local' | 08:05 |
| SvenKieske | didn't check if that actually changed | 08:05 |
| mmalchuk | which means 127.0.01, ::001, and local connect | 08:06 |
| mnasiadka | mmalchuk: so I proposed we do Require ::1 127.0.0.1 - just like tripleo | 08:06 |
| SvenKieske | but that's really a sidetrack, currently. | 08:06 |
| mnasiadka | instead of local | 08:06 |
| mmalchuk | lol) | 08:06 |
| SvenKieske | didn't really investigate if it maybe even was a local deployment issue | 08:06 |
| mmalchuk | 'require local' only in horizon.conf | 08:06 |
| mmalchuk | other services not configured at all | 08:06 |
| SvenKieske | always good to have a productive conversation with you guys :) | 08:07 |
| mmalchuk | so your solution add to all services virtualhost configuration the location /server-status with require 127.0.0.1 ? | 08:08 |
| mnasiadka | well, Debian/Ubuntu default is require local, that's true | 08:08 |
| mnasiadka | mmalchuk: we're having a discussion, aren't we? | 08:08 |
| mmalchuk | sure | 08:08 |
| mmalchuk | lets find the way | 08:08 |
| mnasiadka | instead of doing weird config on haproxy level, it would be better to allow users to configure /server-status properly | 08:09 |
| mmalchuk | I propose the solution wich works | 08:09 |
| mnasiadka | and default to something security sane | 08:09 |
| SvenKieske | agreed to both of you | 08:09 |
| mmalchuk | why it weird? | 08:09 |
| mmalchuk | haproxy is front, it should defend too! | 08:10 |
| mnasiadka | stop those exclamation marks please | 08:10 |
| mnasiadka | shouting is not polite | 08:10 |
| mmalchuk | imho we need many more other deny on front of web services | 08:10 |
| mnasiadka | why I don't think it's a good idea? we're changing role defaults on multiple services, but not all - we're not checking in CI if /server-status is even accessible, we're only fixing security by obscurity on haproxy level (and not everybody is using haproxy) | 08:12 |
| mmalchuk | I'm polite, sorry if this bother someone | 08:12 |
| * SvenKieske is currently in a meeting | 08:12 | |
| mnasiadka | I agree this might fix it for some people, but I'm not convinced it's THE fix | 08:13 |
| SvenKieske | just one thought: we should do security in depth, imho, so disable/secure things at haproxy level, but also on another level, e.g. if haproxy is not used | 08:13 |
| mmalchuk | about haproxyless, this affects only horizon | 08:13 |
| mnasiadka | and while we merge this - it will not automatically fix on all exposed environment, you still need to run deploy ;-) | 08:13 |
| mmalchuk | it can be fixed too, I've planed later | 08:13 |
| mmalchuk | we do upgrades | 08:14 |
| mmalchuk | even on the same releases | 08:14 |
| mmalchuk | in production | 08:14 |
| mnasiadka | Just saying I don't see the urgency on merging this, and surely I'm not happy with having N followups, where each of them needs to be backported all the way back to Yoga | 08:15 |
| mmalchuk | understand of no urgency | 08:15 |
| mnasiadka | Let's continue the discussion on the meeting in the afternoon | 08:15 |
| mmalchuk | I'm only want to close the issue | 08:15 |
| mmalchuk | the way you choose | 08:15 |
| mmalchuk | ok, continue on the meeting | 08:16 |
| SvenKieske | I'll also be in a meeting when the meeting takes place, will try to be somewhat available though. | 08:17 |
| SvenKieske | meetingception | 08:17 |
| mnasiadka | Well, it's not for me to choose - but I prefer to close it once in the way we don't need to attend it anymore and we backport one patch to all stable branches | 08:17 |
| SvenKieske | sure, that would be optimal | 08:19 |
| frickler | SvenKieske: how again is /server-status showing PII? | 08:37 |
| SvenKieske | last time I looked server-status show all visiting clients ip addresses which is PII under gdpr? | 08:38 |
| SvenKieske | it has been some years since I last looked at server-status page, so my information might be outdated, but I doubt it. | 08:39 |
| SvenKieske | see e.g. https://blog.sucuri.net/2012/10/popular-sites-with-apache-server-status-enabled.html | 08:40 |
| mmalchuk | outdated, all sites are fixes | 08:41 |
| mmalchuk | outdated, all sites are fixed | 08:41 |
| SvenKieske | this is really not very new information, unless apache did rewrite that mod_status, which I doubt :) | 08:42 |
| frickler | oh, _visiting_ clients. well I think we're lucky that behind haproxy that's only haproxy itself and it doesn't show the forwarded_for info afaict | 08:42 |
| SvenKieske | ah okay, if it's only the haproxy IP that would be great :) | 08:42 |
| SvenKieske | but yeah, didn't check that, aren't forwared for headers set these days by default? | 08:42 |
| mmalchuk | x-forwared-for used | 08:43 |
| mmalchuk | and the list would contain all clients IP | 08:43 |
| frickler | but x-forwarded-for isn't shown in the status page, at least in my installation | 08:45 |
| frickler | but still leaking a lot of internal information, so should be fixed anyway | 08:45 |
| mmalchuk | there show the real ip address in the list | 08:56 |
| SvenKieske | mmalchuk: are you saying you're not only seeing haproxy ip and the header is set? | 08:58 |
| mmalchuk | I see my real IP from internet along as other clients | 08:59 |
| SvenKieske | mhm, interesting, wondering where your setup differs from ours. | 09:07 |
| frickler | seems the bug is here: https://opendev.org/openstack/kolla-ansible/src/branch/master/ansible/roles/horizon/templates/horizon.conf.j2#L29-L31 | 09:10 |
| frickler | this overrides the restriction for /server-status from status.conf | 09:10 |
| frickler | so it has nothing to do with haproxy really | 09:12 |
| SvenKieske | you gotta love those > 500loc changes where no meaningful review can take place: https://opendev.org/openstack/kolla-ansible/commit/5137f6b35bf264967fc1d676aa18d5b5465c7d13 | 09:14 |
| frickler | well big tasks require big patches, I don't see how you could split adding a 500loc config file into multiple steps | 09:18 |
| SvenKieske | weird, was disconnected from all three IRC networks where I'm online. | 09:31 |
| SvenKieske | ah my provider just decided to reconnect me in the middle of the day..nice | 09:32 |
| opendevreview | Maksim Malchuk proposed openstack/kolla-ansible master: Deny access to public /server-status in http Openstack services https://review.opendev.org/c/openstack/kolla-ansible/+/888943 | 09:36 |
| mmalchuk | frickler I know about the bug you mentioned, updated fix | 09:37 |
| mmalchuk | this also will fix the issue with haproxyless installations where only horizon is affected | 09:37 |
| mmalchuk | SvenKieske agree with your comment about exclamation mark. in my culture it used not for "shouting" | 09:39 |
| mmalchuk | I'm really sorry, but it added automatically how learned in school | 09:40 |
| SvenKieske | that's the trouble we get in an international environment I guess, but we all learn something new everyday :) | 09:48 |
| mnasiadka | no strong emotions needed :) | 09:49 |
| mmalchuk | I will try not use exclamation mark at all, but sometimes I can forgot, so sorry again) | 09:51 |
| mmalchuk | mnasiadka I've did a quick research - we cant use 'require ip 127.0.0.1' because we use api_interface_address to listen in apache, 127.0.0.1 not used at all, and this will completely disable access to server-status. so the haproxy only can block access. also added the fix for horizon in haproxyless configurations. | 09:54 |
| mmalchuk | also update the commit message and release note. | 09:54 |
| mmalchuk | https://review.opendev.org/c/openstack/kolla-ansible/+/888943/7..8 | 09:55 |
| opendevreview | Jan Gutter proposed openstack/kolla-ansible master: WIP debug kolla-ansible-ubuntu-upgrade-cephadm https://review.opendev.org/c/openstack/kolla-ansible/+/890822 | 09:55 |
| opendevreview | Jan Gutter proposed openstack/kolla-ansible master: etcd: Add support for more scenarios https://review.opendev.org/c/openstack/kolla-ansible/+/888012 | 09:57 |
| opendevreview | Jan Gutter proposed openstack/kolla-ansible master: etcd: deduplicate environments for containers https://review.opendev.org/c/openstack/kolla-ansible/+/890208 | 09:57 |
| opendevreview | Jan Gutter proposed openstack/kolla-ansible master: WIP debug kolla-ansible-ubuntu-upgrade-cephadm https://review.opendev.org/c/openstack/kolla-ansible/+/890822 | 09:57 |
| jangutter | I found something interesting: looks like the `ara` logs have been broken for a while now. I know how to fix them, _but_ ara comes with a bit of an extra overhead. | 10:02 |
| jangutter | I find ara to be incredibly useful, but, it's definitely not a universal opinion. | 10:02 |
| opendevreview | Jan Gutter proposed openstack/kolla-ansible master: WIP debug kolla-ansible-ubuntu-upgrade-cephadm https://review.opendev.org/c/openstack/kolla-ansible/+/890822 | 10:57 |
| yusufgungor | Hi, is there any kolla cinder docker image which already has the purestorage pip module? We are getting the error below: | 10:58 |
| yusufgungor | 2023-08-09 13:38:59.407 55 ERROR cinder.volume.manager cinder.volume.drivers.pure.PureDriverException: Missing 'purestorage' python module, ensure the library is installed and available. | 10:58 |
| mnasiadka | jangutter: we disabled generating ara logs long time ago, because we were hitting out of space issues, you can try to enable them back, as long as we won't hit that problem anymore ;-) | 11:03 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: systemd: handle running container without systemd unit https://review.opendev.org/c/openstack/kolla-ansible/+/890198 | 11:04 |
| jangutter | mnasiadka: Oho, yeah, they can go big. Adding the sqlite output uses about 6M, but rendering to html uses 309M | 11:05 |
| mnasiadka | maybe sqlite output and a script to render html on your own laptop | 11:06 |
| jangutter | yeah, I think that's probably the best. | 11:09 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: DNM: Test with docker_restart_policy default https://review.opendev.org/c/openstack/kolla-ansible/+/890658 | 11:27 |
| mmalchuk | yusufgungor no support yet | 11:35 |
| mmalchuk | yusufgungor https://review.opendev.org/c/openstack/kolla-ansible/+/879844 | 11:35 |
| mmalchuk | yusufgungor the change is on review | 11:35 |
| mmalchuk | yusufgungor but some review were merged | 11:37 |
| mmalchuk | yusufgungor https://review.opendev.org/c/openstack/kolla-ansible/+/863298 | 11:37 |
| mmalchuk | yusufgungor https://review.opendev.org/c/openstack/kolla-ansible/+/841453 | 11:37 |
| mmalchuk | yusufgungor if you have an issue please create bug-report | 11:38 |
| yusufgungor | Thanks for reply @mmalchuk I mean we have to install purestorage module into the cinder-volume container like "pip install purestorage" as kolla-ansible documentation states: | 11:44 |
| yusufgungor | "The use of this backend requires that the purestorage SDK package is installed in the cinder-volume container. To do this follow the steps outlined in the kolla image building guide particularly the Package Customisation and Custom Repos sections." | 11:44 |
| yusufgungor | Is it possible to have a docker image version with already installed this module? | 11:44 |
| mmalchuk | yes, create the bug-report and someone will do this | 11:45 |
| mnasiadka | or create a bug report and fix it in Kolla (contributions welcome) | 11:45 |
| mmalchuk | yep | 11:46 |
| SvenKieske | afaik this is a proprietary storage solution, I think that might be the reason why there is not that much support for it? | 11:55 |
| mmalchuk | afaik good storage, thought to buy it | 12:04 |
| mmalchuk | but bought huawei dorado) | 12:05 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Add support for LetsEncrypt-managed certs https://review.opendev.org/c/openstack/kolla-ansible/+/741340 | 12:13 |
| mnasiadka | patchset 223 | 12:16 |
| mnasiadka | I think it's going for an all time record | 12:16 |
| jangutter | mnasiadka: looks like there's enough space to render it, but I think the sqlite solution is the best bang for the buck. https://3228bf3e0a1d8c81580e-c67b93b429db84769e417a4684ceffa6.ssl.cf1.rackcdn.com/890822/6/check/kolla-ansible-ubuntu-upgrade-cephadm/fce6f57/primary/ara-report/ara-html/ | 12:30 |
| mnasiadka | jangutter: seems we have craploads of space looking at https://3228bf3e0a1d8c81580e-c67b93b429db84769e417a4684ceffa6.ssl.cf1.rackcdn.com/890822/6/check/kolla-ansible-ubuntu-upgrade-cephadm/fce6f57/primary/logs/system_logs/df.txt | 12:31 |
| mnasiadka | frickler: ^^ - is it now standard to get 80GB disk? | 12:31 |
| frickler | we should not put support for any proprietary solution into our containers | 12:33 |
| frickler | mnasiadka: it may depend on the provider, let me check what guarantees we have | 12:34 |
| mnasiadka | frickler: I know we get a second disk on some providers - looking at https://github.com/openstack/kolla/blob/7f12d216dc4de2c8d32291c3d6223185ecf2b510/tests/playbooks/pre.yml#L44 | 12:35 |
| jangutter | yusufgungor: can you build your own kolla images? If so, you can add the extra plugin using a config file and the method described here: https://docs.openstack.org/kolla/latest/admin/image-building.html#plugin-functionality | 12:36 |
| frickler | "There is at least 80GB of disk available." but not necessarily on a single disk, yes. https://docs.opendev.org/opendev/infra-manual/latest/testing.html | 12:36 |
| mnasiadka | so we might want to handle the additional disk in kolla-ansible CI as well for /var/lib/docker - that would make more space on / for things like ara HTML files | 12:38 |
| frickler | we might remount/bindmount the disk if there is one, yes | 12:38 |
| jangutter | yusufgungor: as far as I can tell from the docs, it looks like that the purestorage portion you need can't be bundled upstream and has to be added in by the user. | 12:39 |
| frickler | but also make sure not to generate too much log volume. like 300M html output for each job should certainly be avoided | 12:39 |
| mnasiadka | right, true | 12:40 |
| mnasiadka | jangutter: let's go w sqlite then and some script/instructions how to generate ARA HTML | 12:40 |
| jangutter | ack, I'll work up a patch and a readme.txt to put beside the sqlite file. | 12:41 |
| mnasiadka | frickler: I think I finally managed to get to a working state with https://review.opendev.org/c/openstack/kolla-ansible/+/890198/46 (tested in https://review.opendev.org/c/openstack/kolla-ansible/+/890658/14) - would be grateful if you could take a look | 12:41 |
| frickler | 46 revisions in 8 days is also not bad ;) added to my list but likely won't get to it today | 12:43 |
| mnasiadka | frickler: well, first 20 revisions was to understand the problem, next 15 revisions was to get to current state, then I had a ,,better'' idea to use dc.wait, but it didn't work :) | 12:49 |
| frickler | mnasiadka: yes, I just wanted to put that into relation with the "223 iterations for LE in 3 years" above ;) | 12:52 |
| mnasiadka | frickler: true :) | 12:55 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Add support for LetsEncrypt-managed certs https://review.opendev.org/c/openstack/kolla-ansible/+/741340 | 12:56 |
| mnasiadka | damn, forgot | 13:00 |
| mnasiadka | mgoddard mnasiadka hrw bbezak frickler kevko SvenKieske mmalchuk gkoper jangutter - meeting now | 13:00 |
| mnasiadka | #startmeeting kolla | 13:00 |
| opendevmeet | Meeting started Wed Aug 9 13:00:31 2023 UTC and is due to finish in 60 minutes. The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot. | 13:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 13:00 |
| opendevmeet | The meeting name has been set to 'kolla' | 13:00 |
| mnasiadka | #topic rollcall | 13:00 |
| mnasiadka | o/ | 13:00 |
| mmalchuk | \o | 13:00 |
| jangutter | o/ | 13:00 |
| mattcrees | o/ | 13:00 |
| ihalomi | \o | 13:01 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Add support for LetsEncrypt-managed certs https://review.opendev.org/c/openstack/kolla-ansible/+/741340 | 13:01 |
| SvenKieske | o/ | 13:01 |
| SvenKieske | fantastic timing, meeting ended early, right into the next one | 13:01 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: Move to Debian 12 'bookworm' https://review.opendev.org/c/openstack/kolla/+/886088 | 13:02 |
| mnasiadka | #topic agenda | 13:03 |
| mnasiadka | * CI status | 13:03 |
| mnasiadka | * Release tasks | 13:03 |
| mnasiadka | * Regular stable releases (first meeting in a month) | 13:03 |
| mnasiadka | * Current cycle planning | 13:03 |
| mnasiadka | * Additional agenda (from whiteboard) | 13:03 |
| mnasiadka | * Open discussion | 13:03 |
| mnasiadka | #topic CI status | 13:03 |
| mnasiadka | So, kolla and kolla-ansible sound green-ish, magnum jobs are failing due to some designate issue | 13:03 |
| mnasiadka | kayobe upgrade jobs still red - fixing in https://review.opendev.org/c/openstack/kolla-ansible/+/890198/46 | 13:03 |
| mnasiadka | #topic Release tasks | 13:03 |
| mnasiadka | It's R-8 week | 13:04 |
| mnasiadka | we have https://docs.openstack.org/kolla/latest/contributor/release-management.html#r-8-switch-images-to-current-release | 13:04 |
| mnasiadka | Anybody wants to handle switching RDO and UCA to current in-development release? | 13:05 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/yoga: Add documentation for migrating from CS8 to RL9 https://review.opendev.org/c/openstack/kolla-ansible/+/884858 | 13:05 |
| frickler | I wonder whether we still need to do that at all | 13:05 |
| mnasiadka | we use some packages from RDO for sure | 13:06 |
| mnasiadka | I can handle RDO | 13:06 |
| frickler | 2023.2 should work fine with ceph+libvirt from 2023.1 | 13:06 |
| mnasiadka | that true, but OVN users would be happy to get newer OVN version | 13:07 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Add support for LetsEncrypt-managed certs https://review.opendev.org/c/openstack/kolla-ansible/+/741340 | 13:08 |
| frickler | is OVN in UCA? I don't know about RDO | 13:08 |
| SvenKieske | I guess I could maybe take a stab at the UCA part, never did it, but the linked change looks easy enough | 13:08 |
| mnasiadka | frickler: yes it is | 13:08 |
| mnasiadka | ok | 13:08 |
| SvenKieske | don't know anything about RDO "delorean" | 13:08 |
| mnasiadka | #action mnasiadka handle RDO switch to bobcat | 13:08 |
| mnasiadka | #action SvenKieske handle UCA switch to bobcat | 13:08 |
| mnasiadka | #topic Regular stable releases (first meeting in a month) | 13:09 |
| mnasiadka | We agreed last time to do it after the systemd fix gets backported to 2023.1 | 13:09 |
| mnasiadka | it's still not merged in master, so let's try again next week | 13:09 |
| mnasiadka | #topic Current cycle planning | 13:10 |
| mnasiadka | I see good progress on Let's Encrypt | 13:10 |
| mnasiadka | ihalomi: sorry, I didn't have time to have a look in Rocky failures on podman | 13:10 |
| mnasiadka | ihalomi: any other issues on podman patches? Is there something we could review? | 13:10 |
| mnasiadka | (as in merge before we fix the rocky problem) | 13:10 |
| ihalomi | mnasiadka: no other issues, this is basically this is the last obstacle to getting +1 | 13:11 |
| SvenKieske | last time I looked patches looked pretty good, afaik I did a full review, will have another look | 13:11 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/yoga: ovsdpdk: add libdpdk-dev https://review.opendev.org/c/openstack/kolla/+/880317 | 13:12 |
| mnasiadka | ok, I'll have a look too | 13:12 |
| mnasiadka | #topic Additional agenda (from whiteboard) | 13:12 |
| mnasiadka | octavia jobboard patch seems to get better, I'll have a look at it later | 13:13 |
| mnasiadka | debian bookworm support - I'll fix the conflicts in the kolla patch and it seems it's good to merge | 13:13 |
| frickler | I added the precheck verification as discussed | 13:13 |
| mnasiadka | and then we could have a crack at the kolla-ansible side | 13:13 |
| frickler | we need to wait for the haproxy option fixes for bookworm | 13:13 |
| mnasiadka | ok, there's a series of patches | 13:14 |
| mnasiadka | I've seen a proposal to squash them together | 13:14 |
| frickler | I reviewed one patch but didn't track responses | 13:14 |
| frickler | yes, that was me | 13:14 |
| mnasiadka | but maybe it's just easier to merge them as is when they pass properly? | 13:14 |
| mnasiadka | but yes, we need to have a look at pushing those forward, I think the author is not going to work on them promptly enough ;-) | 13:15 |
| frickler | I can have a look again, but it seemed too complicated to me | 13:15 |
| mnasiadka | mattcrees - rabbitmq enable ha queues by default is ready for review: https://review.opendev.org/c/openstack/kolla-ansible/+/882825 | 13:15 |
| mattcrees | Basically I've added support in the CI upgrade jobs to migrate the queue types, and have proposed a KA command to reset rabbit state as this is a key part of the process | 13:16 |
| mnasiadka | ok then, another thing to review queue ;) | 13:17 |
| frickler | ack | 13:17 |
| mnasiadka | ok then | 13:18 |
| mnasiadka | #topic Open discussion | 13:18 |
| SvenKieske | could people with +2 powers take a look at the backports of: https://review.opendev.org/c/openstack/kolla-ansible/+/889189 ? thank you | 13:18 |
| mnasiadka | Do we need to discuss the server-status thing again? | 13:18 |
| SvenKieske | should be straight forward | 13:18 |
| SvenKieske | I _guess_ the current server-status patch now looks fine? :) | 13:18 |
| mmalchuk | mnasiadka I can give an update | 13:19 |
| mmalchuk | this can't be fixed by 'require ip 127.0.0.1' because of https://opendev.org/openstack/kolla-ansible/src/branch/master/ansible/roles/horizon/templates/horizon.conf.j2#L16 | 13:20 |
| mmalchuk | there is already 'require local' - https://httpd.apache.org/docs/2.4/mod/mod_authz_host.html | 13:20 |
| mmalchuk | so accept - the client address matches 127.0.0.0/8, the client address is ::1, both the client and the server address of the connection are the same | 13:20 |
| mmalchuk | last case - is haproxy | 13:20 |
| mmalchuk | I've updated change: https://review.opendev.org/c/openstack/kolla-ansible/+/888943/7..8 | 13:21 |
| mmalchuk | add more info and update release note | 13:21 |
| mnasiadka | Well, I'm still not convinced - Rocky/CentOS have mod_status enabled, but not configured - and it does not show up there. | 13:21 |
| mmalchuk | I'm ready to add mod_status to Rocky/Centos | 13:22 |
| frickler | with no config, the handler isn't used | 13:22 |
| mnasiadka | So if we would remove the default configuration for mod_status in Debian/Ubuntu - we would get to the same state? | 13:22 |
| mmalchuk | will propose Kolla fix | 13:22 |
| mnasiadka | And then on top of this, we could add a feature to configure that properly in kolla-ansible? | 13:22 |
| mnasiadka | Or is my thinking wrong? | 13:23 |
| mmalchuk | frickler wrong | 13:23 |
| SvenKieske | imho your thinking is only "wrong" that it breaks existing users | 13:23 |
| mmalchuk | it can't be configured because of haproxy | 13:23 |
| mmalchuk | add 'require 127.0.0.1' will completely disable status | 13:23 |
| mmalchuk | because we use internal vip for apache listen | 13:24 |
| frickler | mmalchuk: I was talking about why the issue isn't seen in rocky | 13:24 |
| mnasiadka | SvenKieske: it's not a feature, it's a bug for now :) | 13:24 |
| SvenKieske | i.e. removing the default config and adding a role to properly configuring this would require us to backport a completely new role, no? and it must match the old behaviour at least for internal usage | 13:24 |
| mnasiadka | an unplanned feature is a bug from my perspective | 13:24 |
| mmalchuk | frickler did you see last change? https://review.opendev.org/c/openstack/kolla-ansible/+/888943/7..8 | 13:24 |
| mmalchuk | and Rocky is not affeced | 13:24 |
| mmalchuk | this is in release note | 13:25 |
| frickler | yes, I was replying to mnasiadka | 13:25 |
| SvenKieske | mnasiadka: that's really not helpful humor here: it's a security bug because it's reachable from potential untrusted networks on deb/ubuntu; it's a feature there because you can monitor apache stuff from internal networks, I don't want to break those users, do you? | 13:25 |
| mnasiadka | SvenKieske: it's a feature only on Debian/Ubuntu, and it's a Debian/Ubuntu package and default config feature, not a kolla-ansible feature :) | 13:26 |
| SvenKieske | or we backport completely new roles to old releases to fix this, could also work, but imho backports should be minimal, which mmalchuks change really is, there's not much to argue there from the maintenance perspective imho. | 13:26 |
| mmalchuk | mnasiadka I will add the 'feature' to the Rocky/Centos but this is not related to bug | 13:26 |
| mnasiadka | It's not a feature. | 13:27 |
| mmalchuk | we will got the same behavior | 13:27 |
| mmalchuk | on all installations | 13:27 |
| SvenKieske | mnasiadka: do you really want to argue that we only promise to not break users if we built a feature in k-a ourselves? because than I'm out of this discussion, this is ridiculous | 13:28 |
| mnasiadka | It's like with this ansible-core breakage, because we used handlers with import_tasks - bug was rejected because nobody ever planned this usage. | 13:28 |
| mmalchuk | Centos/Redhat users can use monitoring localay, but will not be affected by bug we close now | 13:28 |
| SvenKieske | server-status is a clear apache feature, it's not a bug per se, you just need to care how you enable that | 13:28 |
| mnasiadka | We don't enable tht. | 13:29 |
| mnasiadka | *that | 13:29 |
| mnasiadka | It's enabled by default | 13:29 |
| mnasiadka | only on 50% of the distributions we support | 13:29 |
| mmalchuk | we enable that in horizon) | 13:29 |
| SvenKieske | yeah sure, but don't break the default then | 13:29 |
| mnasiadka | From my perspective it's a Debian/Ubuntu packaging bug | 13:29 |
| mmalchuk | https://opendev.org/openstack/kolla-ansible/src/branch/master/ansible/roles/horizon/templates/horizon.conf.j2#L30 | 13:29 |
| frickler | yes, and it is safe by default, adding haproxy in front is what breaks things | 13:29 |
| SvenKieske | "only on 50%"..I'm out | 13:29 |
| frickler | and adding haproxy is what kolla does | 13:29 |
| SvenKieske | your not having a honests helpful discussion imho | 13:30 |
| mmalchuk | without this server status not available even without haproxy | 13:30 |
| mnasiadka | do we really need to override Location / in horizon.conf? | 13:30 |
| frickler | mnasiadka: that's actually a second issue | 13:30 |
| mnasiadka | frickler: which could fix the first issue | 13:30 |
| mmalchuk | mnasiadka Ubuntu/Debian enable more modules than Centos/Rocky not only mod_status | 13:31 |
| frickler | mnasiadka: no | 13:31 |
| mnasiadka | second thing - I'm all in deny'ing /server-status in haproxy on ALL frontends | 13:31 |
| mnasiadka | but the patch denies it only on some of the services | 13:31 |
| mnasiadka | and doesn't include a CI task that will check if we don't encounter the same bug in future | 13:31 |
| mnasiadka | are we going to iterate on this bug every time it shows up? | 13:31 |
| SvenKieske | I honestly don't care _how_ we do this. I care about two things: a) fix the security bug (I don't care whoever introduced it, upstream, we, little green men from mars) b) preserve legitimate use cases like internal monitoring without breaking them | 13:32 |
| mmalchuk | on some - because of apache2 only services need this | 13:32 |
| frickler | ok, but then we agree on doing the filtering as proposed in haproxy? | 13:32 |
| SvenKieske | I'm baffled that needs to be argued, like, at all. | 13:32 |
| mmalchuk | we didnt use external vip on CI - this is another issue | 13:33 |
| mnasiadka | Seems we have no other option, that to do the filtering as proposed in haproxy - question what about users that use external load balancers, should we update the docs for them? | 13:33 |
| SvenKieske | "question what about users that use external load balancers" <- can you elaborate what you mean by that? I don't understand | 13:33 |
| frickler | if the loadbalancer isn't on the same host as apache, only the fix in the horizon location statement is needed | 13:33 |
| mmalchuk | imho there are no such users | 13:34 |
| mnasiadka | mmalchuk: of course there are | 13:34 |
| frickler | there are, but not affected | 13:34 |
| mnasiadka | thanks for clarification instead of denying the problem ;-) | 13:34 |
| frickler | those won't match "Require local" | 13:34 |
| mnasiadka | right | 13:34 |
| mmalchuk | server-status on cinder api ? for example? kidding | 13:34 |
| mnasiadka | but now, with the current shape of the patch - we enable mod_status on CentOS and Rocky for Horizon, right? | 13:35 |
| SvenKieske | you mean people who don't use haproxy but use a different LB? well they need to do the filtering themselves I guess? a doc patch would be nice, but not strictly necessary imho | 13:35 |
| mmalchuk | right | 13:35 |
| mmalchuk | i will propose patch | 13:35 |
| SvenKieske | they need to provide all balancing config themselves either way, so I hope they know what they are doing I guess. | 13:36 |
| mmalchuk | we just add handler to config in kolla | 13:36 |
| mmalchuk | should we backport this 'feature' ? | 13:36 |
| SvenKieske | it's really funny that we are back to the original solution again (are we? I have lost track..) | 13:36 |
| mnasiadka | mmalchuk: I'm asking, because https://review.opendev.org/c/openstack/kolla-ansible/+/888943/8/ansible/roles/horizon/templates/horizon.conf.j2#34 will change behaviour on CentOS/Rocky | 13:37 |
| SvenKieske | well if it breaks existing users without the backport I'd argue we need a backport | 13:37 |
| mnasiadka | if it's planned - then include that in the reno, but for backporting - we shouldn't probably doing that | 13:37 |
| mmalchuk | frickler as said 'not affected' | 13:37 |
| mmalchuk | there is 'require local' already in apache config from package | 13:37 |
| mnasiadka | not in CentOS/Rocky | 13:38 |
| mnasiadka | and here you add it for ALL distributions | 13:38 |
| frickler | mnasiadka: this will not enable server-status | 13:38 |
| mnasiadka | are you sure? | 13:38 |
| mmalchuk | not. but there is no handler there! | 13:38 |
| frickler | yes | 13:38 |
| mmalchuk | sorry) | 13:38 |
| frickler | exactly, no handler | 13:38 |
| mnasiadka | ah ok, SetHandler is missing | 13:38 |
| mmalchuk | so centos/redhat not affected at all | 13:38 |
| mmalchuk | even with fix | 13:38 |
| SvenKieske | yeah this only requires a local ip to get to the route, but if apache has no route to that path everything is fine, I didn't check rocky myself | 13:39 |
| mmalchuk | but wen I propose change to Kolla container - there will be work, so we need merge first fix | 13:39 |
| mmalchuk | when | 13:39 |
| SvenKieske | I wish the "enable this on rocky" would be a separate discussion, I have no eggs in the basket regarding rocky :) | 13:39 |
| frickler | I would skip that part unless rocky users ask for it | 13:40 |
| mmalchuk | I will propose, lets people decide | 13:41 |
| mmalchuk | we will have same behaviour in all installations | 13:41 |
| mnasiadka | that's another discussion | 13:42 |
| mmalchuk | yep | 13:42 |
| SvenKieske | yeah from a strict technical standpoint we should try to provide similar functionality for all distros | 13:42 |
| mmalchuk | indeed | 13:42 |
| SvenKieske | afaik that's why we use upstream packages, maybe we should package apache httpd ourselves? /troll | 13:42 |
| mmalchuk | it can be described in the docs. about 'curl... server-status' to check | 13:42 |
| mmalchuk | huge overhead | 13:43 |
| jangutter | do we have 2 minutes for a quick report about our favourite etcd version? (timeboxed) | 13:43 |
| mnasiadka | SvenKieske: better tell me why Debian/Ubuntu exposes all users to that ,,issue'' | 13:43 |
| mmalchuk | the answer 'because')) | 13:44 |
| SvenKieske | just debian things I guess, not a fan of the tech side - I was always a fedora guy personally - I stumbled upon this error in a different context some years ago. | 13:44 |
| mmalchuk | because they want | 13:44 |
| frickler | jangutter: imo later is better, but we can ony bump minor version by one per cycle? | 13:44 |
| jangutter | TL;DR etcd 3.4 drops deprecated API interfaces. Nearly every kolla-ansible service uses the old interface. | 13:45 |
| frickler | also please stop these distro wars | 13:45 |
| mmalchuk | they enable huge list of modules and expose more info | 13:45 |
| SvenKieske | debian has the philosophy of enabling everything to nanny the user (autostarting services), except where it's useful (providing auditd with remote support).. | 13:45 |
| mnasiadka | frickler: not a distro war, but it should be semi-secure by default ;) | 13:45 |
| frickler | jangutter: oh, so 3.4 is too new for us? | 13:45 |
| mnasiadka | jangutter: so all services need a change in their config | 13:46 |
| mnasiadka | jangutter: ? | 13:46 |
| SvenKieske | I can work with every distro, and there are objectively bad and good things in all of them | 13:46 |
| jangutter | and Zun is particularly badly affected - they depend on a particularly painful combination of docker and old etcd. I only see WIP things for it, but it requires some hard rework for them. | 13:46 |
| frickler | jangutter: afaict devstack is also hardcoded to 3.3, guess we need to bump there first | 13:46 |
| jangutter | Cinder (and hopefully all tooz affected things) are configurable and "works" on 3.4. | 13:46 |
| mnasiadka | jangutter: we can drop zun support, it's been painful already ;) | 13:46 |
| SvenKieske | uh that's a bummer, should there be a blueprint or ML post to coordinate upgrades for etcd then? | 13:47 |
| jangutter | Yah, noting that 3.3 is no longer maintained, and 3.4 is likely to be soon, sooner migration is better. | 13:48 |
| frickler | I'll propose a bump in devstack and see what happens there | 13:48 |
| frickler | and then we can check in kolla again | 13:49 |
| jangutter | There's a lot of "v3alpha" hardcoded everywhere in clients, especially python ones. | 13:49 |
| mnasiadka | ok, so that seems like a longer topic - maybe worth a thread on ML? | 13:49 |
| jangutter | (I got it to work in k-a, trying to fix the unrelated noise in the job) | 13:49 |
| jangutter | I'll send out a post! | 13:49 |
| opendevreview | Maksim Malchuk proposed openstack/kolla-ansible master: Deny access to public /server-status in http Openstack services https://review.opendev.org/c/openstack/kolla-ansible/+/888943 | 13:50 |
| frickler | I have two more small patches: https://review.opendev.org/c/openstack/kolla-ansible/+/871054 and https://review.opendev.org/c/openstack/kolla-ansible/+/876650 | 13:50 |
| jangutter | Zun might be a casualty unfortunately, they hit a deprecated "feature" in docker _and_ etcd. | 13:50 |
| mnasiadka | I don't think it's overly active, so no hard feelings | 13:51 |
| mnasiadka | don't know if there's a lot of users | 13:51 |
| jangutter | Thanks! end of timebox, feel free to ping me if you're interested or have Zun contacts | 13:51 |
| mnasiadka | ok | 13:52 |
| mnasiadka | anybody anything else? | 13:52 |
| mmalchuk | kayobe reviews? | 13:53 |
| mnasiadka | once kayobe CI is fixed we can handle that | 13:53 |
| mnasiadka | currently it's broken due to systemd issue | 13:54 |
| mmalchuk | https://review.opendev.org/c/openstack/kayobe/+/879554 | 13:54 |
| mmalchuk | https://review.opendev.org/c/openstack/kayobe/+/861397 | 13:54 |
| mmalchuk | please these two | 13:54 |
| mnasiadka | they won't pass the CI, nor the gate, upgrade jobs are broken | 13:54 |
| mnasiadka | ok, I think we've had the longest meeting this year | 13:55 |
| mmalchuk | already passed | 13:55 |
| mnasiadka | thanks for coming | 13:55 |
| mmalchuk | thank you | 13:55 |
| mnasiadka | mmalchuk: but they won't pass now. | 13:55 |
| mnasiadka | #endmeeting | 13:55 |
| opendevmeet | Meeting ended Wed Aug 9 13:55:20 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 13:55 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/kolla/2023/kolla.2023-08-09-13.00.html | 13:55 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/kolla/2023/kolla.2023-08-09-13.00.txt | 13:55 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/kolla/2023/kolla.2023-08-09-13.00.log.html | 13:55 |
| SvenKieske | thank you for chairing :) | 13:55 |
| frickler | \o | 13:55 |
| * SvenKieske going to the next meeting.. | 13:55 | |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/2023.1: rabbitmq: Fix repo for ubuntu aarch64 https://review.opendev.org/c/openstack/kolla/+/887177 | 15:58 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/yoga: Use erlang-25 from copr on aarch64 https://review.opendev.org/c/openstack/kolla/+/886966 | 15:58 |
| mmalchuk | folks, please review and merge https://review.opendev.org/c/openstack/kolla-ansible/+/888943 | 16:15 |
| opendevreview | Merged openstack/kolla-ansible master: rabbitmq: add rabbitmq_datadir_volume parameter https://review.opendev.org/c/openstack/kolla-ansible/+/876650 | 16:22 |
| opendevreview | Merged openstack/kolla-ansible master: ironic: add ironic_agent_files_directory parameter https://review.opendev.org/c/openstack/kolla-ansible/+/871054 | 16:34 |
| opendevreview | Michal Arbet proposed openstack/kolla master: Rework letsencrypt https://review.opendev.org/c/openstack/kolla/+/887347 | 17:12 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Add support for LetsEncrypt-managed certs https://review.opendev.org/c/openstack/kolla-ansible/+/741340 | 18:22 |
| opendevreview | Maksim Malchuk proposed openstack/kolla-ansible master: Deny access to public /server-status in http Openstack services https://review.opendev.org/c/openstack/kolla-ansible/+/888943 | 19:57 |
| opendevreview | Maksim Malchuk proposed openstack/kolla-ansible master: [DNM] test haproxy external vip https://review.opendev.org/c/openstack/kolla-ansible/+/890758 | 19:57 |
| opendevreview | Maksim Malchuk proposed openstack/kolla-ansible master: Deny access to public /server-status in http Openstack services https://review.opendev.org/c/openstack/kolla-ansible/+/888943 | 19:58 |
| opendevreview | Maksim Malchuk proposed openstack/kolla-ansible master: [DNM] test haproxy external vip https://review.opendev.org/c/openstack/kolla-ansible/+/890758 | 19:58 |
| opendevreview | Gaël THEROND proposed openstack/kolla-ansible master: Improve designate role support. https://review.opendev.org/c/openstack/kolla-ansible/+/878270 | 20:03 |
| opendevreview | Michal Arbet proposed openstack/kolla master: Rework letsencrypt https://review.opendev.org/c/openstack/kolla/+/887347 | 20:18 |
| opendevreview | Michal Arbet proposed openstack/kolla master: Rework letsencrypt https://review.opendev.org/c/openstack/kolla/+/887347 | 20:24 |
| opendevreview | Maksim Malchuk proposed openstack/kolla-ansible master: Deny access to public /server-status in http Openstack services https://review.opendev.org/c/openstack/kolla-ansible/+/888943 | 20:52 |
| opendevreview | Maksim Malchuk proposed openstack/kolla-ansible master: [DNM] test haproxy external vip https://review.opendev.org/c/openstack/kolla-ansible/+/890758 | 20:55 |
| opendevreview | Maksim Malchuk proposed openstack/kolla-ansible master: Deny access to public /server-status in http Openstack services https://review.opendev.org/c/openstack/kolla-ansible/+/888943 | 21:18 |
| opendevreview | Maksim Malchuk proposed openstack/kolla-ansible master: [DNM] test haproxy external vip https://review.opendev.org/c/openstack/kolla-ansible/+/890758 | 21:19 |
| opendevreview | Gaël THEROND proposed openstack/kolla-ansible master: Improve designate role support. https://review.opendev.org/c/openstack/kolla-ansible/+/878270 | 23:01 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!