| mnasiadka | SvenKieske: well, redhat also introduced those version checks | 06:46 |
|---|---|---|
| mnasiadka | SvenKieske: let's agree to one thing, if we start chasing people for wording in commit messages - they are not going to keep contributing - and if you post only nit: type of comments - you add +2 anyway | 06:51 |
| mnasiadka | (or if you have some OCD on commit messages - then just help the author and edit it by yourself before merging) ;-) | 06:51 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: Remove setuptools pin https://review.opendev.org/c/openstack/kolla/+/901828 | 06:56 |
| SvenKieske | ack | 07:17 |
| SvenKieske | morning | 07:17 |
| SvenKieske | if you talked about the cert stuff, the wording was not just wording but a little bit unclear imho, but then again I shouldn't +2 those then I guess, if it's not only wording, fair point. | 07:18 |
| SvenKieske | anyone here has opinions on the pre-commit/CI mail from Stephen? I'm intrigued by ruff myself, could be a nice little speedup for our CI: https://opendev.org/openstack/osc-lib/src/branch/master/.pre-commit-config.yaml the mail: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/thread/Q5M7RGCSSFHFIK3RA7MTSY4K6KKOECNE/ | 07:29 |
| SvenKieske | could also reduce our test-requirements.txt if we move to pre-commit, generally I guess it would reduce the maintenance burden (a little bit) in the long run. | 07:30 |
| mnasiadka | SvenKieske: I meant https://review.opendev.org/c/openstack/kolla-ansible/+/928325?tab=comments | 07:34 |
| mnasiadka | SvenKieske: I've seen mail from Stephen, it might bring us something good, but I think we can try doing that next cycle | 07:35 |
| SvenKieske | but I didn't +2 that one? :) | 07:37 |
| SvenKieske | maybe I misunderstood you? | 07:37 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: systemd: Add Wants=docker.service for docker https://review.opendev.org/c/openstack/kolla-ansible/+/929218 | 07:38 |
| mnasiadka | Well, you commented on wording in commit message - and both are really nits from my perspective | 07:38 |
| mnasiadka | we don't usually block on that, because it's a bit irritating for the authors | 07:38 |
| SvenKieske | okay, I can agree on that. | 07:38 |
| mnasiadka | it's not linux kernel :) | 07:38 |
| SvenKieske | But I was explicit why I was not voting just yet, which was not the nits. But agreed, I can either fix those myself or not talk about them. I can understand the reasoning, having received far to many nits myself on my own reviews :D | 07:39 |
| mnasiadka | I've added my +2, so now you can either update the commit message and +2+w, or live with that commit message and merge it :D | 07:40 |
| SvenKieske | in that regard I found seans comment on the ML thread insightful that he was weared down working on upstream bugs because of the only comments he received were about formatting. | 07:40 |
| SvenKieske | so I really should find my branch again where I was testing black code formatter. :D | 07:41 |
| mnasiadka | since it's friday - anybody interested in https://review.opendev.org/c/openstack/kolla-ansible/+/922420 ? ;-) | 07:44 |
| SvenKieske | we have API timeouts in the cephadm jobs, not even reaching our attempt=10 loop counter :( https://zuul.opendev.org/t/openstack/build/2233d6086d9148ba8a4cb97f8f9d4a29/log/primary/logs/ansible/test-core-openstack | 07:46 |
| SvenKieske | in test-core-openstack.sh when simply deleting volumes | 07:46 |
| SvenKieske | mnasiadka: interesting, I see the usecase for quickly starting it, did you also use it to stop everything? does that work without crashing something? | 07:48 |
| SvenKieske | I guess my real question is: do all service roles include the common role, that is: what exactly are "_all_ kolla containers"? | 07:49 |
| mnasiadka | I did stop and start - it's not fast due to all those exit 139/143 errors which I'm trying to fix in some $free time ;-) | 07:49 |
| mnasiadka | SvenKieske: we have that for some time (the errors), but they are not manifesting every time, probably this is the next thing somebody should look at in terms of cephadm jobs stability | 07:50 |
| SvenKieske | agreed, reminds me of the general openstack api slowness :( | 07:51 |
| mnasiadka | Request to http://192.0.2.10:5000/v3/auth/tokens timed out | 07:51 |
| mnasiadka | that's more interesting | 07:51 |
| mnasiadka | but that also means we need better error handling, or just retrying even if the command fails | 07:52 |
| SvenKieske | yes, I guess that should do the trick | 08:00 |
| opendevreview | Pierre Riteau proposed openstack/kolla stable/2023.2: Cap ansible.posix below 1.6.0 https://review.opendev.org/c/openstack/kolla/+/929219 | 08:02 |
| opendevreview | Mark Goddard proposed openstack/kolla-ansible master: prometheus: Support overriding address of scrape targets https://review.opendev.org/c/openstack/kolla-ansible/+/899615 | 08:09 |
| opendevreview | Pierre Riteau proposed openstack/kolla stable/2023.2: Cap ansible.posix below 1.6.0 https://review.opendev.org/c/openstack/kolla/+/929219 | 08:32 |
| opendevreview | Merged openstack/kolla master: openvswitch: Drop ovs_ensure_configured.sh https://review.opendev.org/c/openstack/kolla/+/907901 | 08:33 |
| opendevreview | Verification of a change to openstack/kolla master failed: rabbitmq: Switch to Cloudsmith mirrors https://review.opendev.org/c/openstack/kolla/+/928928 | 08:51 |
| opendevreview | Roman Krček proposed openstack/kolla-ansible master: Refactor nova upgrade task https://review.opendev.org/c/openstack/kolla-ansible/+/928325 | 09:07 |
| opendevreview | Pierre Riteau proposed openstack/kolla stable/2023.1: Cap ansible.posix below 1.6.0 https://review.opendev.org/c/openstack/kolla/+/929234 | 10:27 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: Remove setuptools pin https://review.opendev.org/c/openstack/kolla/+/901828 | 11:00 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: Remove setuptools pin https://review.opendev.org/c/openstack/kolla/+/901828 | 11:00 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Add parameter to always merge config into proxyql https://review.opendev.org/c/openstack/kolla-ansible/+/928611 | 11:02 |
| kevko | ^^ anybody ? | 11:08 |
| opendevreview | Dawud proposed openstack/kolla-ansible master: Add size limits to Fluentd buffers https://review.opendev.org/c/openstack/kolla-ansible/+/924359 | 11:19 |
| opendevreview | Pierre Riteau proposed openstack/kolla-ansible stable/2023.1: Fix checkouts of previous release repositories https://review.opendev.org/c/openstack/kolla-ansible/+/929261 | 11:22 |
| opendevreview | Pierre Riteau proposed openstack/kolla-ansible stable/2023.1: CI: Fix checkouts of previous release repositories https://review.opendev.org/c/openstack/kolla-ansible/+/929261 | 11:39 |
| kevko | Thanks | 12:06 |
| SvenKieske | FYI: kevko, mnasiadka: stable/2023.2 pipelines fail now due to https://review.opendev.org/c/openstack/kolla/+/929219 missing, which fails itself, investigating.. | 12:40 |
| SvenKieske | seems like an unrelated mirror issue during build, at least that's what I see when looking at the build log. rechecking now | 12:47 |
| opendevreview | Merged openstack/kolla master: rabbitmq: Switch to Cloudsmith mirrors https://review.opendev.org/c/openstack/kolla/+/928928 | 12:48 |
| SvenKieske | ansible version foo is always so entertaining :) | 12:58 |
| priteau | SvenKieske: I have replied to your comment on https://review.opendev.org/c/openstack/kolla-ansible/+/929143 | 13:14 |
| priteau | I would like to backport it to fix all the stable branches which are hard to fix otherwise (cross dependencies between kolla and kolla-ansible) | 13:14 |
| opendevreview | Merged openstack/kolla-ansible master: Refactor nova upgrade task https://review.opendev.org/c/openstack/kolla-ansible/+/928325 | 13:26 |
| SvenKieske | priteau: replied, but I'm a little bit unsure about the backporting, because it could be argued this is user visible and existing tools might rely on output and error messages to surface at the same place :-/ | 13:49 |
| priteau | Would it help if we displayed both streams in case of JSON parsing error? | 14:00 |
| priteau | Although it would be tricky because we have lost the respective ordering of the two streams. | 14:01 |
| SvenKieske | maybe frickler has input on this, if we parse this output somewhere downstream. | 14:02 |
| priteau | Note that any data on stderr would likely break JSON parsing (unless the stderr was valid JSON lines that were somehow produced exactly in the right place), so your kolla-ansible run would already be failing before this change | 14:04 |
| priteau | We are giving it a chance to keep running if the output is valid JSON | 14:05 |
| SvenKieske | yeah, I'm more concerned with some bash script just grepping for an error reason and then doing stuff, something like that. | 14:05 |
| priteau | Honestly I really want to fix the stable branches and this is the easiest path for it, we could even revert it once this is done if there is a general agreement that we don't want this change. | 14:07 |
| priteau | Well, another approach is to drop a whole bunch of CI jobs, merge the ansible.posix fixes and re-add them | 14:10 |
| SvenKieske | sorry I'm currently busy with another task. I actually don't understand why this is strictly needed to merge the posix fix? does the json parser error let the CI pipeline error as well? | 14:17 |
| priteau | It's because of a chain of dependencies: ansible.posix fix needs to go in 2023.1 first, then 2023.2 (because of upgrade jobs), but zed->2023.1 jobs are also broken because we are using the wrong kolla-ansible version there (fixed in https://review.opendev.org/c/openstack/kolla-ansible/+/929261) | 14:25 |
| priteau | And there are cross-repo dependencies between the fixes in kolla and the fixes in kolla-ansible | 14:25 |
| priteau | So this is why I was suggesting to backport the stdout/stderr fix everywhere, which would make Zuul happy and we can merge all the other fixes | 14:26 |
| SvenKieske | so I think I understand the dependency problems with the other patches you mentioned, but I fail to see how they need this stderr fix? What's the technical reason for that? Isn't that just a nice to have thing? | 14:29 |
| priteau | I think stdout/stderr is a nice to have fix, which as a side effect would fix CI | 14:30 |
| priteau | But to fix CI we need it in all affected branches | 14:32 |
| priteau | I backported to yoga, technically I think only zed might be required | 14:32 |
| SvenKieske | I don't think how the output separation fixes CI? The pin of the posix stuff fixes CI. | 14:32 |
| opendevreview | Will Szumski proposed openstack/kayobe master: Aligns Kayobe Ansible version with Kolla Ansible https://review.opendev.org/c/openstack/kayobe/+/926639 | 14:33 |
| SvenKieske | we also might want to stop testing upgrade jobs from an unmaintained branch, don't you think? | 14:33 |
| priteau | We can also drop zed->2023.1 jobs, that's another option. | 14:33 |
| priteau | Output separation also fixes CI because CI is failing due to the new warning raised by ansible.posix 1.6.0 which breaks JSON parsing. But the module still appears to work if we ignore the warning. | 14:34 |
| SvenKieske | I mean I'm also fine with keeping and fixing them, but technically it would be correct to drop them. We might want to keep them for ensuring people on these old releases can actually still upgrade. I guess some downstreams care about those (stackhpc and kevko, mmalchuk) afaik | 14:35 |
| SvenKieske | ah that was just a warning, I thought that was a hard error, thx for the explanation | 14:35 |
| priteau | I would be for keeping them if they're not too much effort. | 14:35 |
| SvenKieske | it's weird that they only warn about it, if they "remove support" though..lol | 14:35 |
| SvenKieske | ok I guess you convinced me :) as this seems to unblock CI I'll bite. | 14:38 |
| priteau | As I said earlier, if you change your mind we can also revert, after all the CI fixes are merged. | 14:38 |
| priteau | Are you OK to give W+1 to the master patch? | 14:39 |
| SvenKieske | done | 14:39 |
| priteau | Thanks | 14:39 |
| SvenKieske | is that WIP on purpose? https://review.opendev.org/c/openstack/kolla-ansible/+/929148 | 14:39 |
| priteau | Yes, it was a test backport | 14:40 |
| priteau | I will update it | 14:40 |
| SvenKieske | ok | 14:40 |
| priteau | But we have to go in reverse order anyway | 14:40 |
| priteau | yoga, zed, 2023.1, 2023.2 | 14:40 |
| SvenKieske | ok, makes sense | 14:40 |
| mnasiadka | well, should we drop the upgrade jobs from unmaintained branches? | 14:52 |
| mnasiadka | or at least mark them as non voting? | 14:52 |
| mnasiadka | We're not going to maintain the unmaintained forever, will we? | 14:52 |
| opendevreview | Merged openstack/kolla-ansible master: systemd: Add Wants=docker.service for docker https://review.opendev.org/c/openstack/kolla-ansible/+/929218 | 14:54 |
| opendevreview | Merged openstack/kolla-ansible master: Add parameter to always merge config into proxyql https://review.opendev.org/c/openstack/kolla-ansible/+/928611 | 14:54 |
| SvenKieske | mnasiadka: marking as non voting is a good compromise imho. don't you guys still support zed based stuff? :) would be nice to know your upgrades still work? personally I don't care for these old releases, but I can empathize with the users :D | 14:56 |
| mnasiadka | zed was always a ,,jump'' release for us | 14:56 |
| mnasiadka | we did yoga, then upgraded customers to antelope, and now we do SLURP to caracal | 14:56 |
| mnasiadka | so basically - I would say we mark that as non voting | 14:57 |
| mnasiadka | we care for the maintained branches, if there's a volunteer to push patches to unmaintained and gain attraction of core reviewers to merge that - fine by me | 14:57 |
| SvenKieske | ok | 15:02 |
| greatgatsby | Hello. We're using kolla-ansible, looking like we're going to need to modify sentinel down-after-milliseconds config for redis-sentinel. Can I just confirm that the kolla-ansible redis role doesn't seem to use merge_configs like most other roles? We're wondering if we need to patch kolla-ansible to change the default redis config for sentinel down-after-milliseconds? Thanks for any suggestions. | 15:06 |
| SvenKieske | ah you asked that a few days ago alrady | 15:09 |
| SvenKieske | for what exactly are you using redis sentinel? afaik not all openstack services support that, e.g. keystonemiddleware is still missing https://review.opendev.org/c/openstack/keystonemiddleware/+/915872 | 15:10 |
| SvenKieske | the redis stuff is under heavy patching, currently, see this relation chain: https://review.opendev.org/c/openstack/kolla-ansible/+/927104 | 15:11 |
| opendevreview | Pierre Riteau proposed openstack/kolla-ansible stable/2023.1: Separate outputs of kolla_toolbox inner module https://review.opendev.org/c/openstack/kolla-ansible/+/929148 | 15:12 |
| opendevreview | Pierre Riteau proposed openstack/kolla-ansible stable/2023.2: Separate outputs of kolla_toolbox inner module https://review.opendev.org/c/openstack/kolla-ansible/+/929294 | 15:12 |
| opendevreview | Pierre Riteau proposed openstack/kolla-ansible stable/2024.1: Separate outputs of kolla_toolbox inner module https://review.opendev.org/c/openstack/kolla-ansible/+/929295 | 15:13 |
| SvenKieske | greatgatsby: if that is configured in the regular redis.conf, that is templated since the beginning and no config merge is being done, so you would need to adjust the template redis-sentinel.conf.j2, yes. | 15:14 |
| SvenKieske | feel free to provide a patch I guess :) | 15:15 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: [release] Change RDO's DELOREAN_DEPS repo to Dalmatian https://review.opendev.org/c/openstack/kolla/+/929296 | 15:21 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/2024.1: systemd: Add Wants=docker.service for docker https://review.opendev.org/c/openstack/kolla-ansible/+/929297 | 15:22 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/2023.2: systemd: Add Wants=docker.service for docker https://review.opendev.org/c/openstack/kolla-ansible/+/929298 | 15:23 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/2023.1: systemd: Add Wants=docker.service for docker https://review.opendev.org/c/openstack/kolla-ansible/+/929299 | 15:23 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/2023.1: systemd: Add Wants=docker.service for docker https://review.opendev.org/c/openstack/kolla-ansible/+/929299 | 15:24 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/2023.1: systemd: Add Wants=docker.service for docker https://review.opendev.org/c/openstack/kolla-ansible/+/929299 | 15:25 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/2023.1: systemd: Add Wants=docker.service for docker https://review.opendev.org/c/openstack/kolla-ansible/+/929299 | 15:25 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/2023.1: systemd: Add Wants=docker.service for docker https://review.opendev.org/c/openstack/kolla-ansible/+/929299 | 15:25 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/2023.1: systemd: Add Wants=docker.service for docker https://review.opendev.org/c/openstack/kolla-ansible/+/929299 | 15:26 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/2023.1: systemd: Add Wants=docker.service for docker https://review.opendev.org/c/openstack/kolla-ansible/+/929299 | 15:26 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/2023.1: systemd: Add Wants=docker.service for docker https://review.opendev.org/c/openstack/kolla-ansible/+/929299 | 15:26 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/2023.1: systemd: Add Wants=docker.service for docker https://review.opendev.org/c/openstack/kolla-ansible/+/929299 | 15:27 |
| mnasiadka | that's how it ends when you want to quickly edit something in the web editor... | 15:27 |
| SvenKieske | stop it! :D | 15:27 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/2024.1: rabbitmq: Switch to Cloudsmith mirrors https://review.opendev.org/c/openstack/kolla/+/929300 | 15:28 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/2023.2: rabbitmq: Switch to Cloudsmith mirrors https://review.opendev.org/c/openstack/kolla/+/929301 | 15:29 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/2023.2: rabbitmq: Switch to Cloudsmith mirrors https://review.opendev.org/c/openstack/kolla/+/929301 | 15:30 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/2023.1: rabbitmq: Switch to Cloudsmith mirrors https://review.opendev.org/c/openstack/kolla/+/929302 | 15:31 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/2023.1: rabbitmq: Switch to Cloudsmith mirrors https://review.opendev.org/c/openstack/kolla/+/929302 | 15:35 |
| opendevreview | Will Szumski proposed openstack/kayobe master: kolla-openstack: Add ironic-inspector dir to config https://review.opendev.org/c/openstack/kayobe/+/924163 | 15:35 |
| mnasiadka | ok then, enough | 15:35 |
| opendevreview | Pierre Riteau proposed openstack/kayobe master: Support skipping SSH keyscan for older switch devices https://review.opendev.org/c/openstack/kayobe/+/926547 | 15:36 |
| greatgatsby | SvenKieske: thanks. We enabled redis when setting up designate. The designate docs specifically say "If multiple nodes are assigned to be Designate workers, then you must enable a supported coordination backend, currently only redis is supported." | 15:44 |
| johnsom | There are a few options for Designate, but Redis is a good one. | 15:45 |
| greatgatsby | well is seems enabling redis has gotten us into some trouble. The default sentinel failover time is too low for our environment, so sentinel is basically stuck in a failover loop, which has cascading effects. And if I understand SvenKieske correctly, enabling redis is not really 100% supported? | 15:48 |
| opendevreview | Will Szumski proposed openstack/kayobe master: Add internal vip address to no_proxy https://review.opendev.org/c/openstack/kayobe/+/922787 | 15:48 |
| johnsom | It is in Designate, but I can't answer for Kolla | 15:48 |
| greatgatsby | thanks, that's good to know. I think we just need to get redis/redis-sentinel stable for now. Unfortunately our dev environment is much smaller than prod, so we sometimes get burned by scaling issues. | 15:51 |
| opendevreview | Pierre Riteau proposed openstack/kayobe master: kolla-openstack: Add ironic-inspector dir to config https://review.opendev.org/c/openstack/kayobe/+/924163 | 15:51 |
| greatgatsby | would it be advisable for us to reconfig our cluster to only use redis for designate? | 15:54 |
| johnsom | Well, Octavia can use Redis as well. For example, Red Hat OpenStack is shipping with both Designate and Octavia using Redis. | 15:55 |
| greatgatsby | also good to know. I had planned on seeing how other deployment types were configuring/using redis. | 15:56 |
| greatgatsby | really appreciate all the feedback, just trying to achieve the most stable environment for our workloads | 16:02 |
| SvenKieske | yeah many services support redis, also in kolla, but some don't yet in kolla, or don't support TLS, if you need that. you have to check the services themselves. unfortunately right now you also need to rewrite the config I guess, as there is only a single template afaics. | 16:03 |
| opendevreview | Merged openstack/kolla-ansible master: Separate outputs of kolla_toolbox inner module https://review.opendev.org/c/openstack/kolla-ansible/+/929143 | 16:04 |
| johnsom | There are longer term discussions happening about moving to valkey due to the Redis licensing issues, but that is just in the discussion phase now. There will need to be oslo work to start that process. | 16:06 |
| opendevreview | Will Szumski proposed openstack/kayobe master: Add internal vip address to no_proxy https://review.opendev.org/c/openstack/kayobe/+/922787 | 16:15 |
| opendevreview | Will Szumski proposed openstack/kolla-ansible master: Add NET_RAW to dnsmasq container https://review.opendev.org/c/openstack/kolla-ansible/+/910501 | 16:44 |
| opendevreview | Will Szumski proposed openstack/kolla-ansible master: Add NET_RAW to dnsmasq container https://review.opendev.org/c/openstack/kolla-ansible/+/910501 | 16:46 |
| opendevreview | Merged openstack/kayobe master: Add support for customising Neutron physical network names https://review.opendev.org/c/openstack/kayobe/+/922335 | 16:53 |
| opendevreview | Merged openstack/kayobe master: Prevent running from a different Kayobe configuration repository https://review.opendev.org/c/openstack/kayobe/+/903139 | 17:45 |
| atmark | hello, I'm on Yoga release ( I know it's EOL ) and I plan to migrate RabbitMQ from transient to durable queues by following this instruction https://docs.openstack.org/kolla-ansible/zed/reference/message-queues/rabbitmq.html#high-availability . My question is with `Step 1. Stop all OpenStack services which use RabbitMQ ` - does stopping neutron and nova containers impact | 18:19 |
| atmark | running VMs? | 18:19 |
| kevko | Nope | 18:23 |
| kevko | Api and openstack as a service | 18:23 |
| kevko | K8s will be problem on openstack | 18:24 |
| atmark | kevko: what do you mean by "Api and openstack as a service"? | 18:26 |
| atmark | I don't any VM running k8s | 18:26 |
| atmark | I don't have* | 18:27 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Add ansible-core as a dependency https://review.opendev.org/c/openstack/kolla-ansible/+/922369 | 18:33 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Add ansible-core as a dependency https://review.opendev.org/c/openstack/kolla-ansible/+/922369 | 18:34 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Do not remove inventory file if placed in /etc/kolla https://review.opendev.org/c/openstack/kolla-ansible/+/908429 | 19:37 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!