| *** mmalchuk_ is now known as mmalchuk | 03:44 | |
| opendevreview | Verification of a change to openstack/kolla-ansible master failed: etcd: Add support for more scenarios https://review.opendev.org/c/openstack/kolla-ansible/+/888012 | 06:31 |
|---|---|---|
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: magnum: support kubeconfig configuration file https://review.opendev.org/c/openstack/kolla-ansible/+/896292 | 07:13 |
| opendevreview | Piotr Parczewski proposed openstack/kolla-ansible master: Adjust Ceph metrics scrape interval in Prometheus https://review.opendev.org/c/openstack/kolla-ansible/+/902129 | 07:30 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Adjust Ceph metrics scrape interval in Prometheus https://review.opendev.org/c/openstack/kolla-ansible/+/902129 | 07:37 |
| * frickler cries a bit about unstable ci | 08:08 | |
| frickler | mnasiadka: I don't think we need anything from https://review.opendev.org/q/project:openstack/ansible-collection-kolla+status:open , so we could approve the release patch for that? | 08:14 |
| mnasiadka | frickler: makes sense - feel free to approve :) | 08:15 |
| frickler | done | 08:22 |
| mnasiadka | https://review.opendev.org/c/openstack/kolla/+/902053 and https://review.opendev.org/c/openstack/kolla/+/896644 ready for reviews | 08:40 |
| opendevreview | Verification of a change to openstack/kolla-ansible master failed: etcd: Add support for more scenarios https://review.opendev.org/c/openstack/kolla-ansible/+/888012 | 08:42 |
| mnasiadka | what happened with Debuntu upgrade... | 08:58 |
| mnasiadka | https://1091c07913eb4632aa6d-ee11d96724f7ceb5cf79a550bea264c0.ssl.cf5.rackcdn.com/888012/11/gate/kolla-ansible-ubuntu-upgrade/9c7b5dd/primary/logs/kolla/all-CRITICAL.txt | 09:00 |
| mnasiadka | https://22999be8fea712a10cfd-df24963d5172a658f3710dd49321d0e5.ssl.cf1.rackcdn.com/888012/11/gate/kolla-ansible-debian-upgrade/8ac994d/primary/logs/kolla/all-CRITICAL.txt | 09:01 |
| mnasiadka | identical | 09:01 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Add oslo_db.exception.DBConnectionError to check-logs.sh https://review.opendev.org/c/openstack/kolla-ansible/+/902131 | 09:04 |
| mnasiadka | frickler: ^^ | 09:04 |
| mnasiadka | funny it only hits in gate | 09:12 |
| opendevreview | Mark Goddard proposed openstack/kayobe master: CI: Run configuration dump against localhost https://review.opendev.org/c/openstack/kayobe/+/902056 | 09:25 |
| opendevreview | Merged openstack/kolla-ansible stable/2023.1: Fix Horizon WSGI application log parsing https://review.opendev.org/c/openstack/kolla-ansible/+/902066 | 09:31 |
| opendevreview | Merged openstack/kolla-ansible stable/zed: Fix Horizon WSGI application log parsing https://review.opendev.org/c/openstack/kolla-ansible/+/902069 | 09:31 |
| opendevreview | Merged openstack/kolla-ansible stable/yoga: Fix Horizon WSGI application log parsing https://review.opendev.org/c/openstack/kolla-ansible/+/902070 | 09:31 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Fail on fluentd log parsing errors https://review.opendev.org/c/openstack/kolla-ansible/+/742399 | 09:42 |
| opendevreview | Merged openstack/kolla master: Add magnum-cluster-api driver to Kolla images https://review.opendev.org/c/openstack/kolla/+/902101 | 10:15 |
| opendevreview | Merged openstack/kolla master: Handle the case where docker is not available https://review.opendev.org/c/openstack/kolla/+/902053 | 10:19 |
| opendevreview | Merged openstack/kolla master: Drop docker-py from requirements.txt https://review.opendev.org/c/openstack/kolla/+/896644 | 10:19 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: bump td-agent lts from v4 to v5 https://review.opendev.org/c/openstack/kolla/+/894948 | 10:51 |
| opendevreview | Merged openstack/kolla-ansible master: etcd: Add support for more scenarios https://review.opendev.org/c/openstack/kolla-ansible/+/888012 | 11:13 |
| mnasiadka | frickler: I see neutron is special - some other upgrade job failure - https://4a9836736643be5ed458-c393df920c3ba3d74686e2bffd0842f4.ssl.cf5.rackcdn.com/902131/1/check/kolla-ansible-ubuntu-upgrade-cephadm/ab3f02a/primary/logs/kolla/all-CRITICAL.txt | 11:18 |
| mnasiadka | jangutter: https://review.opendev.org/q/topic:%22update-etcd-v3.4%22 - I see only merge conflicts - can you fix? | 11:18 |
| jangutter | yep, on it | 11:19 |
| mnasiadka | thanks | 11:22 |
| opendevreview | Jan Gutter proposed openstack/kolla-ansible master: Update etcd docs and reno https://review.opendev.org/c/openstack/kolla-ansible/+/902086 | 11:24 |
| opendevreview | Jan Gutter proposed openstack/kolla-ansible master: etcd: deduplicate environments for containers https://review.opendev.org/c/openstack/kolla-ansible/+/890208 | 11:24 |
| opendevreview | Jan Gutter proposed openstack/kolla-ansible master: etcd: update to v3.4 https://review.opendev.org/c/openstack/kolla-ansible/+/890465 | 11:24 |
| guesswhat[m] | mnasiadka: what do I need to change here https://review.opendev.org/c/openstack/kolla-ansible/+/888496 for getting Approval ? | 11:27 |
| opendevreview | Merged openstack/kayobe master: Deduplicate before picking merge strategy https://review.opendev.org/c/openstack/kayobe/+/901336 | 11:37 |
| sylvr | jovial: here's the log from dhcpdump for dnsmasq and for dhcpd https://pastebin.com/q3en5DMD https://pastebin.com/wf8MLRuh | 12:05 |
| sylvr | no filename are provided by the dnsmasq's dhcp, seems odd | 12:07 |
| opendevreview | Jan Gutter proposed openstack/kolla-ansible master: Update etcd docs and reno https://review.opendev.org/c/openstack/kolla-ansible/+/902086 | 12:08 |
| opendevreview | Jan Gutter proposed openstack/kolla-ansible master: etcd: deduplicate environments for containers https://review.opendev.org/c/openstack/kolla-ansible/+/890208 | 12:08 |
| opendevreview | Jan Gutter proposed openstack/kolla-ansible master: etcd: update to v3.4 https://review.opendev.org/c/openstack/kolla-ansible/+/890465 | 12:08 |
| sylvr | and with dhcpd I don't think I manged to pass the correct file since the machines keep retrying PXE boot | 12:12 |
| jovial | slyvr: thanks for that. It does seem odd... Possibly enabling iPXE could help? | 12:12 |
| frickler | I wanted to handle the etcd merge conflicts once the base patch is merged, seems the CI didn't agree with me :-( | 12:13 |
| sylvr | jovial: I'm not sure I can enable iPXE for my client, or is it something to enable in Ironic/Bifrost ? | 12:14 |
| jovial | slyvr: I think to do that you may need to set `enabled_boot_interfaces: "ipxe,pxe"` in `$KAYOBE_CONFIG_PATH/kolla/config/bifrost/bifrost.yml`. I noticed in the dnsmaq config you posted was missing the iPXE related config options. To apply that, use: `kayobe seed service deploy`. | 12:16 |
| sylvr | okay, this line isn't present in the config file, can I set it in kayobe ? | 12:20 |
| jovial | yes, just create the file: `$KAYOBE_CONFIG_PATH/kolla/config/bifrost/bifrost.yml` and pop it in there | 12:21 |
| sylvr | the file exist so that's good | 12:21 |
| sylvr | okay so I'm running the kayobe command hoping this will fix things | 12:22 |
| jovial | slyvr: looking at your dnsmasq config again, it kind of looks like you are missing this whole section: https://github.com/openstack/bifrost/blob/e41ed498324e527400b50897fbb8b51ac909db1e/playbooks/roles/bifrost-ironic-install/templates/dnsmasq.conf.j2#L96-L119 | 12:27 |
| jovial | which might suggest that `enable_inspector_discovery` is set to false | 12:28 |
| jovial | Does anything show up with `grep -R "enable_inspector_discovery" $KOLLA_CONFIG_PATH`? | 12:29 |
| sylvr | grep -Rnw 'ostack/src/kayobe-config' -e 'enable_inspector_discovery' => ostack/src/kayobe-config/etc/kolla/config/bifrost/bifrost.yml:8:enable_inspector_discovery: true | 12:29 |
| jovial | OK, that looks good | 12:29 |
| sylvr | even the config path ? | 12:30 |
| opendevreview | Jan Gutter proposed openstack/kolla-ansible master: etcd: deduplicate environments for containers https://review.opendev.org/c/openstack/kolla-ansible/+/890208 | 12:30 |
| opendevreview | Jan Gutter proposed openstack/kolla-ansible master: etcd: update to v3.4 https://review.opendev.org/c/openstack/kolla-ansible/+/890465 | 12:30 |
| jovial | slyvr: Yeah, kayobe generates config for kolla and that is what is populated in ` ostack/src/kayobe-config/etc/kolla/` | 12:31 |
| jovial | never modify files in that directory directly | 12:31 |
| sylvr | because there's still a kolla directory inside the kayobe config (etc/kayobe/kolla/) with the passwords inside | 12:31 |
| sylvr | okay, but how do you set ipxe and pxe then ? in etc/kayobe/bifrost.yml ? | 12:32 |
| jovial | slyvr: Yeah, that is normal. Config in that directory is used by kayobe to generate the the output in ` ostack/src/kayobe-config/etc/kolla/` | 12:32 |
| jovial | it would be in `ostack/src/kayobe-config/etc/kayobe/kolla/config/bifrost/bifrost.yml` | 12:33 |
| opendevreview | Merged openstack/kayobe master: Honor custom ssh args in kayobe-ansible-user.yml https://review.opendev.org/c/openstack/kayobe/+/788276 | 12:33 |
| sylvr | hum... okay | 12:33 |
| sylvr | so, the dnsmasq.conf is still missing the dhcp-boot part | 12:34 |
| sylvr | oh wait | 12:35 |
| sylvr | I modified the wrong file haven't I | 12:35 |
| sylvr | yuup | 12:35 |
| sylvr | shouldn't I modify this file ? : ostack/src/kayobe-config/etc/kayobe/bifrost.yml | 12:36 |
| jovial | Possibly, its weird though.. I'd still expect to see that `dhcp-boot` even with ipxe not in the enabled_boot_interfaces | 12:36 |
| jovial | slyvr: Put it in `ostack/src/kayobe-config/etc/kayobe/kolla/config/bifrost/bifrost.yml` | 12:38 |
| jovial | I'm not sure we have a variable to expose the boot interfaces in etc/kayobe/bifrost.yml | 12:39 |
| sylvr | oh yeah okay | 12:39 |
| sylvr | well, I don't think it worked, I'm going to replay all playbook just to be sure | 12:50 |
| sylvr | (there's still a loop of DHCP discovery and DHCP offer, and the dnsmasq.conf is still missing some lines) | 12:54 |
| opendevreview | Sven Kieske proposed openstack/kolla-ansible master: Add a flag to enable quorum queues https://review.opendev.org/c/openstack/kolla-ansible/+/898543 | 13:31 |
| jovial | slyvr: How does the dnsmasq config file look now? | 13:32 |
| sylvr | jovial : it's the same | 13:33 |
| sylvr | how can I check if the correct files are read for the config ? | 13:34 |
| jovial | slyvr: I wonder if that means that the overrides aren't being used | 13:34 |
| sylvr | yup, that's why I want to be sure that the config is really made from the file I'm modifying | 13:35 |
| sylvr | well, the kolla file contain the correct options | 13:36 |
| sylvr | https://pastebin.com/kvHWfn6C | 13:36 |
| sylvr | at least from what I understand from the files lmao | 13:37 |
| jovial | slyvr: Could you increase the verbosity and look for something like: `Running command: . /home/stack/yoga-rl9-migration/venvs/kolla-ansible/bin/activate && kolla-ansible deploy-bifrost` in the output | 13:38 |
| sylvr | so I had -vvv ? | 13:39 |
| jovial | slyvr: lets see what it is passing as the `--configdir` option | 13:39 |
| jovial | slyvr: yeah, that should do it | 13:40 |
| sylvr | okay ! I'll send the logs, so I run just kayobe seed service deploy ? | 13:40 |
| jovial | yes, please. That would be great. | 13:41 |
| sylvr | okay ! I hope the logfile won't be too big ^^ | 13:41 |
| jovial | You could do something like: `kayobe seed service deploy -kt none -t none -vvv` and that will at least tell us where it is looking for the config (but won't actually do any changes) | 13:43 |
| mnasiadka | bbezak, frickler: fluentd v5 ready - https://review.opendev.org/q/topic:%22fluentd5%22 | 13:44 |
| mmalchuk | mgoddard tls followup ready - https://review.opendev.org/c/openstack/kayobe/+/793697 | 13:45 |
| frickler | mnasiadka: thx, checking. fyi I'm having some flakyness with my broadband connection today, might not respond fast during the meeting | 13:49 |
| mnasiadka | frickler: I'm going to make the meeting short - I'll be in two meetings at the same time | 13:49 |
| frickler | ok | 13:50 |
| mnasiadka | but I think we need to focus on the things we need to merge before we release rc1 | 13:51 |
| mnasiadka | and a final next week | 13:51 |
| sylvr | jovial : running grep "kolla-ansible deploy-bifrost" logs_quick (the file containing the output of the command you sent me) returns nothing | 13:51 |
| mnasiadka | frickler: https://review.opendev.org/c/openstack/kolla-ansible/+/742399 - this will make us fail on fluentd parsing errors - so we don't get into this in the future again - needs second +2 | 13:52 |
| sylvr | I think it might be a good idea to make my config from scratch at this point | 13:52 |
| mnasiadka | mgoddard mnasiadka bbezak frickler kevko SvenKieske mmalchuk gkoper jangutter jsuazo jovial - meeting in 7 minutes | 13:53 |
| sylvr | jovial : I'm going to redo the whole installation/config of kayobe, hopefully I can do things way quicker now that I already have exemple of config, I hope it'll fix issues, I can make my repo public if you'd like to check the configfiles directly | 14:00 |
| jovial | slyvr: Sounds awesome, if I could see the config, that would definitely aid in debugging - thanks :) | 14:01 |
| SvenKieske | mnasiadka: do we want to start? :) | 14:02 |
| mnasiadka | oh crap | 14:02 |
| mnasiadka | yes | 14:02 |
| mnasiadka | #startmeeting kolla | 14:02 |
| opendevmeet | Meeting started Wed Nov 29 14:02:14 2023 UTC and is due to finish in 60 minutes. The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot. | 14:02 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 14:02 |
| opendevmeet | The meeting name has been set to 'kolla' | 14:02 |
| mnasiadka | #topic rollcall | 14:02 |
| mnasiadka | o/ | 14:02 |
| SvenKieske | o/ | 14:02 |
| mattcrees | \o | 14:02 |
| frickler | \o | 14:02 |
| jovial | \o | 14:02 |
| jangutter | \o | 14:02 |
| mhiner | o/ | 14:02 |
| mnasiadka | #topic agenda | 14:04 |
| mnasiadka | * Release tasks | 14:04 |
| mnasiadka | * Open discussion | 14:04 |
| mnasiadka | We make it basically short today - to use the time to review RP+1 patches and RC1 this week | 14:04 |
| mnasiadka | #topic Release tasks | 14:04 |
| mnasiadka | ansible-collection-kolla rc1 is on the way | 14:05 |
| mnasiadka | let's check kolla | 14:05 |
| mnasiadka | Basically fluentd - https://review.opendev.org/q/topic:%22fluentd5%22 | 14:05 |
| mnasiadka | and then we're ready to release rc1 | 14:05 |
| mnasiadka | first add labels for transition in https://review.opendev.org/c/openstack/kolla/+/901508 | 14:06 |
| mnasiadka | second in kolla-ansible - use these labels https://review.opendev.org/c/openstack/kolla-ansible/+/901509 | 14:06 |
| mnasiadka | and then bump version - https://review.opendev.org/c/openstack/kolla/+/894948 | 14:06 |
| mmalchuk | o/ | 14:06 |
| mnasiadka | any questions to those? | 14:06 |
| mnasiadka | ok, I see review comments are coming in - good, will address | 14:08 |
| mnasiadka | what about kolla-ansible - let me check | 14:08 |
| mnasiadka | mainly etcd - https://review.opendev.org/q/topic:%22update-etcd-v3.4%22 | 14:08 |
| mnasiadka | What about quorum queues? https://review.opendev.org/c/openstack/kolla-ansible/+/898543 | 14:09 |
| mnasiadka | Are we merging that now - or waiting for C? | 14:09 |
| mnasiadka | SvenKieske, mattcrees ? | 14:09 |
| mattcrees | I'm currently in the process of testing if the migration strategy we used for HA queues will also work for quorum queues. So far, it seems like it will. (https://docs.openstack.org/kolla-ansible/latest/reference/message-queues/rabbitmq.html#high-availability) | 14:10 |
| SvenKieske | well the cephadm job is unstable with it | 14:10 |
| SvenKieske | but I was not able to figure out why so far. :/ | 14:10 |
| frickler | cephadm is unstable without it, too, I think? | 14:10 |
| SvenKieske | xD yeah that's true, so it might also be unrelated | 14:11 |
| mnasiadka | yeah, we need to put more cycles into that instability | 14:11 |
| mnasiadka | but I've seen that mainly we're failing this job due to errors in neutron-server logs | 14:11 |
| mnasiadka | so it might be some more tweaks to check-logs.sh | 14:11 |
| SvenKieske | I just fixed the Merge conflict, from my pov it could get merged, I only saw people so far who had good experience with quorum queues | 14:11 |
| mnasiadka | ok, let's try to get the etcd stuff in - then we can see if that one can get it | 14:12 |
| mnasiadka | get in | 14:12 |
| SvenKieske | ok | 14:12 |
| mattcrees | If you can give me an hour or so to finish a bit of testing, I'll then be happy to merge it | 14:12 |
| mnasiadka | mattcrees: you can test until tomorrow ;-) | 14:12 |
| mattcrees | I also want to make some follow-up prechecks and docs, but those don't need to block the merge | 14:12 |
| SvenKieske | mattcrees:sure! :) | 14:12 |
| mnasiadka | prechecks and docs can be backported from C I guess | 14:12 |
| mattcrees | Yes that's my thinking too | 14:13 |
| mnasiadka | and the quorum queue support could be backported all the way to Yoga I guess - but without changing any defaults | 14:13 |
| mnasiadka | but that's another story | 14:13 |
| mnasiadka | ok, basically that's it from me | 14:13 |
| mnasiadka | #topic Open discussion | 14:13 |
| mattcrees | Provided the oslo messagine support goes back that far too ;) | 14:13 |
| mnasiadka | mattcrees: I think it does | 14:13 |
| mnasiadka | ok, Open discussion time | 14:14 |
| mnasiadka | ah | 14:14 |
| mnasiadka | jovial: forgot about kayobe - but I think you're chasing that internally in SHPC, right? | 14:14 |
| mnasiadka | and you need to wait for kolla and kolla-ansible to release | 14:14 |
| mnasiadka | to do some pre-branching tasks | 14:14 |
| SvenKieske | is it just me or was CI more unstable the last few weeks? | 14:15 |
| mnasiadka | it's just the CI | 14:15 |
| jovial | Yeah, for kayobe: This is is the list of patches I was hoping to get in: https://review.opendev.org/q/project:openstack/kayobe+status:open+label:Review-Priority%252B1. Let me know if I've missed any important ones. | 14:15 |
| mnasiadka | the javascript error on checking zuul output wasn't helpful as well :) | 14:15 |
| opendevreview | Merged openstack/kolla-ansible master: CI: Add oslo_db.exception.DBConnectionError to check-logs.sh https://review.opendev.org/c/openstack/kolla-ansible/+/902131 | 14:16 |
| opendevreview | Merged openstack/kolla-ansible master: Update etcd docs and reno https://review.opendev.org/c/openstack/kolla-ansible/+/902086 | 14:16 |
| mnasiadka | ok then, off to get those patches merged and checking the CI stability | 14:18 |
| mnasiadka | I mean that's my plan | 14:18 |
| mmalchuk | jovial https://review.opendev.org/c/openstack/kayobe/+/793697 is one of the most important for a 2 years | 14:18 |
| mnasiadka | Let's meet next week with a christmas present for the community - and then work on C | 14:18 |
| mnasiadka | Thanks - and sorry for the short meeting, but I'm overloaded today | 14:19 |
| mnasiadka | #endmeeting | 14:19 |
| opendevmeet | Meeting ended Wed Nov 29 14:19:14 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 14:19 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/kolla/2023/kolla.2023-11-29-14.02.html | 14:19 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/kolla/2023/kolla.2023-11-29-14.02.txt | 14:19 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/kolla/2023/kolla.2023-11-29-14.02.log.html | 14:19 |
| mmalchuk | mnasiadka thanks | 14:19 |
| samcat116 | I have an all in one node deployed with yoga via k-a. Sometimes I'll see openvswitch_vswitchd die for some reason. The container will restart, but won't become healthy as vswitchd will complain that pidfile check failed. Removing the /var/run/openvswitch/ovs-vswitchd.pid file on the host side and manually restarting the container seems to fix it, but is there a way to automatically clean up that file when the container exits? | 14:37 |
| mnasiadka | jangutter: did we break something in cinder-ceph with etcd? https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_93a/890208/11/check/kolla-ansible-ubuntu-cephadm/93a6c6d/primary/logs/kolla/cinder/cinder-volume.txt | 14:38 |
| frickler | mnasiadka: yes, there was a comment earlier, we need the whole series merged to fix that | 14:38 |
| mnasiadka | frickler: fantastic | 14:39 |
| frickler | mnasiadka: well the other option was to revert to bump in kolla and redo later, but I thought just moving forward would be better | 14:39 |
| jangutter | Yeah, apologies, it's a bit of a circular dependency for that job., | 14:40 |
| jangutter | https://review.opendev.org/c/openstack/kolla-ansible/+/890465 should fix it. | 14:41 |
| jangutter | (sorry, need to run, will be back in 2.5h) | 14:41 |
| sylvr | jovial : here's the repository https://forge.univ-lyon1.fr/p1919734/kayobe-config I'm moving some files and config from the old one (but I log very thorougly this time) | 14:52 |
| SvenKieske | mnasiadka: can you have a look at my proposal for the variable definition? any other core would also suffice, I guess :) https://review.opendev.org/c/openstack/kolla-ansible/+/896292 | 15:11 |
| opendevreview | Jake Hutchinson proposed openstack/kolla-ansible stable/yoga: Change regex for the multiline parser at the fluentd rabbitmq input https://review.opendev.org/c/openstack/kolla-ansible/+/901709 | 15:32 |
| opendevreview | Verification of a change to openstack/kolla-ansible master failed: CI: Fail on fluentd log parsing errors https://review.opendev.org/c/openstack/kolla-ansible/+/742399 | 15:35 |
| SvenKieske | mhm, is the gate broken? | 15:45 |
| frickler | hmm ... "ModuleNotFoundError: No module named 'rich'" ... some regression in podman maybe? | 15:47 |
| frickler | just released ... .post1 doesn't sound healthy to me ... https://pypi.org/project/podman/4.8.0.post1/ | 15:49 |
| frickler | maybe we do need to pin versions after all? | 15:50 |
| sylvr | jovial : with the current configuration, nothing changed, I'll check if discovery is enabled (or is it the default ? ) | 15:52 |
| jovial | sylvr: its disabled by default in bifrost, but kayobe should be enabling it. Did you manage to get the verbose output of seed service deploy? | 15:53 |
| jovial | It does seem like kolla isn't using the config files kayobe has generated | 15:54 |
| SvenKieske | frickler: this seems to be the culprit: https://github.com/containers/podman-py/commit/78d57dd116c32b7251c9024c17f3d635d30587ba | 15:55 |
| SvenKieske | frickler: I asked upstream when we implemented this initially, they only support the latest version, basically - I would need to dig up the original review to recall the exact wording, which I saved there | 15:55 |
| sylvr | jovial : oops, I'll do that rn, how do you want me to send you the logs? | 15:55 |
| SvenKieske | it | 15:56 |
| sylvr | jovial : well, I had issue before with dnsmasq not having the same config for the DHCP server... | 15:57 |
| SvenKieske | it's still weird, "rich" is in the requirements.txt of podman-py, so why can't it be imported? | 15:57 |
| SvenKieske | mhm, they also got a pyproject.toml and each file references the other and says dependencies should be updated in both, but that doesn't seem to be the case? https://github.com/containers/podman-py/blob/main/pyproject.toml | 15:58 |
| jovial | sylvr: The logs would be handy. Especially the `Running command` bit when it trys to deploy bifrost. | 15:59 |
| sylvr | well, I'm running kayobe seed service deploy -vvv | tee log_file | 16:00 |
| SvenKieske | mhm, a local installation of podman-py at least works.. | 16:01 |
| SvenKieske | frickler: I filed an issue: https://github.com/containers/podman-py/issues/350 | 16:08 |
| sylvr | jovial : deployment finished, how can I send you the logs ? | 16:14 |
| sylvr | the file is 2.9MB | 16:15 |
| frickler | SvenKieske: "tox -e py3" is a simple reproducer for me locally | 16:23 |
| frickler | and "rich" isn't installed in the tox venv, likely since it is missing from pyproject.toml. as a workaround we can likely add it to test-reqs.txt? | 16:25 |
| SvenKieske | seems sane | 16:26 |
| * frickler proposes a patch to see what else will break | 16:26 | |
| frickler | meh, the installation outside of tox comes from a-c-k (except for debian), that's getting more complicated than I was hoping for | 16:40 |
| mnasiadka | nobody said it will be simple | 16:40 |
| SvenKieske | could it at least be easy? :D | 16:41 |
| mnasiadka | frickler: https://zuul.opendev.org/t/openstack/build/78886ceb45a14d6aa7dd3d32aaedbc2f - what the heck happened here? ;-) | 16:41 |
| SvenKieske | mnasiadka: see comments above, specifically: https://github.com/containers/podman-py/issues/350 | 16:43 |
| opendevreview | Dr. Jens Harbott proposed openstack/kolla-ansible master: podman: install "rich" dependency https://review.opendev.org/c/openstack/kolla-ansible/+/902170 | 16:44 |
| jovial | slyvr: Any way you are happy with. Could email them to me at will@stackhpc.com if you like. | 16:44 |
| frickler | mnasiadka: ^^ might fix it, though I'm too tired to be sure, will check CI results later, feel free to amend if needed | 16:45 |
| mnasiadka | will do | 16:45 |
| SvenKieske | there's a small whitespace missing, otherwise LGTM | 16:45 |
| SvenKieske | err, rather too much whitespace, not missing | 16:45 |
| mnasiadka | that whitespace is not related to this patch, is it? | 16:45 |
| * SvenKieske needs to get some sleep | 16:46 | |
| SvenKieske | ah indeed | 16:46 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Fail on fluentd log parsing errors https://review.opendev.org/c/openstack/kolla-ansible/+/742399 | 16:47 |
| mnasiadka | ok, rebased on the rich patch | 16:47 |
| mnasiadka | SvenKieske: that means we're not linting tox.ini :) | 16:47 |
| SvenKieske | yeah, I won't ask why today, though ;P | 16:47 |
| mnasiadka | every question adds more work :) | 16:48 |
| SvenKieske | so I won't ask :D | 16:48 |
| SvenKieske | https://xkcd.com/1319/ comes to mind :) | 16:49 |
| SvenKieske | at least https://zuul.opendev.org/t/openstack/build/25141e6affa2409b8093fb65f25139e5 looks better now :) | 16:54 |
| frickler | that's the easy part that I could test locally ;) but good to see that confirmed anyway | 16:55 |
| SvenKieske | well I think this should really not break anything else, but you never know.. | 16:56 |
| SvenKieske | I'll guess I call it a day, especially when looking at tomorrows endless list of meetings :) | 16:57 |
| SvenKieske | lol rocky9-podman job failed.. | 16:57 |
| SvenKieske | anyway, o/ | 16:57 |
| mnasiadka | frickler: I guess we should add rich to the list of podman_sdk_pip_packages in a-c-k? | 16:58 |
| frickler | mnasiadka: I was thinking about that, but I'm really hoping podman people will fix their blunder soon enough so that we don't have to | 16:59 |
| frickler | so I'd be fine to just fix our CI for now and then wait until Friday maybe? | 17:00 |
| mnasiadka | podman jobs failed, we would need to override podman_sdk_pip_packages in tests/templates/globals-default.yml.j2 | 17:00 |
| mnasiadka | let me update your patch | 17:00 |
| frickler | yeah, I wasn't sure if the var would make it through from the zuul env, thx | 17:01 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: podman: install "rich" dependency https://review.opendev.org/c/openstack/kolla-ansible/+/902170 | 17:01 |
| mnasiadka | I left the var in zuul just to run those jobs to check | 17:02 |
| mnasiadka | once it passes green I'll remove it | 17:02 |
| mnasiadka | we could pass through the vars in that way so we don't have to manage everything in that file, might be a good idea for C | 17:02 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: podman: install "rich" dependency https://review.opendev.org/c/openstack/kolla-ansible/+/902170 | 17:04 |
| mnasiadka | actually editing globals-default made nearly all jobs to run | 17:04 |
| opendevreview | Merged openstack/kayobe master: kolla-bifrost: Add precheck for inspection allocation pool https://review.opendev.org/c/openstack/kayobe/+/901748 | 17:10 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Fail on fluentd log parsing errors https://review.opendev.org/c/openstack/kolla-ansible/+/742399 | 17:42 |
| opendevreview | Jan Gutter proposed openstack/kolla-ansible master: etcd: deduplicate environments for containers https://review.opendev.org/c/openstack/kolla-ansible/+/890208 | 17:43 |
| opendevreview | Jan Gutter proposed openstack/kolla-ansible master: etcd: update to v3.4 https://review.opendev.org/c/openstack/kolla-ansible/+/890465 | 17:43 |
| opendevreview | Sven Kieske proposed openstack/kolla-ansible master: Add a flag to enable quorum queues https://review.opendev.org/c/openstack/kolla-ansible/+/898543 | 17:48 |
| mnasiadka | frickler: https://review.opendev.org/c/openstack/kolla-ansible/+/901509 - if you could have a look in this one tomorrow - would be nice (kolla-ansible patch between kolla fluentd patches) | 17:53 |
| guesswhat[m] | Do You know that Skyline has an integration to Prometheus and its exporters via query? It has own monitoring center in UI which aggregate common exporters ( libvirt, rabbit, mysql, etc ) | 18:04 |
| mnasiadka | nice, haven't really got time to look into skyline | 18:06 |
| mnasiadka | ugh, kolla also broken by rich | 18:07 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: podman: install "rich" dependency https://review.opendev.org/c/openstack/kolla/+/902179 | 18:07 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: fluentd: Add labels for transition to v5 https://review.opendev.org/c/openstack/kolla/+/901508 | 18:08 |
| guesswhat[m] | mnasiadka: is there any module in kolla which is capable of merging yaml ? would be great to be able to overide skyline.yaml from kolla config dir | 18:08 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: bump td-agent lts from v4 to v5 https://review.opendev.org/c/openstack/kolla/+/894948 | 18:08 |
| mnasiadka | guesswhat[m]: https://github.com/openstack/kolla-ansible/blob/master/ansible/action_plugins/merge_yaml.py - that one? | 18:09 |
| mnasiadka | we use it in multiple roles | 18:09 |
| mnasiadka | ok, enough for today, let's see what zuul says tomorrow morning ;) | 18:10 |
| ozzzo_work | Where can I find a reference showing which openstack version runs on which RHEL version? | 18:31 |
| ozzzo_work | I'm trying to figure out how I can upgrade from Train on RHEL8 to Wallaby on RHEL9. I believe that I have to do the RHEL8->9 upgrade on one of the intermediate steps. Is it Ussuri? | 18:52 |
| guesswhat[m] | is enough to put merge_yaml here https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/skyline/tasks/config.yml#L20C6-L20C6 to enable merging for skyline ? | 19:18 |
| mnasiadka | ozzzo: we don’t support RHEL - as in we don’t test it | 19:18 |
| mnasiadka | ozzzo: but centos 8 to 9 switch is in yoga | 19:19 |
| ozzzo | Is Yoga the only version that supports both 8 and 9? | 19:19 |
| ozzzo | ISTR seeing a document that lays out Centos/Openstack version compatibility but I can't find it now | 19:21 |
| mnasiadka | Yes, Yoga is the version that supports both python 3.6 and 3.9 | 19:24 |
| mnasiadka | Kolla publishes c8s and rocky linux 9 images on quay.io/openstack.kolla if that helps | 19:25 |
| opendevreview | Lukas M proposed openstack/kolla-ansible master: skyline: allow to merge skyline.yaml config with custom config https://review.opendev.org/c/openstack/kolla-ansible/+/902182 | 19:38 |
| guesswhat[m] | mnasiadka: here ^ ( this will allow to configure sso and prometheus integration even if its not yet supported by kolla ) | 19:39 |
| opendevreview | Merged openstack/kolla-ansible master: podman: install "rich" dependency https://review.opendev.org/c/openstack/kolla-ansible/+/902170 | 19:51 |
| dcapone2004 | hi, 2023.1 release....globals.yml does not contain a storage_interface option.... this option was in my older globals.yml deployment...what controls the storage interface used in the deployment now? | 19:55 |
| -opendevstatus- NOTICE: The Gerrit service on review.opendev.org will be restarting momentarily for a patch update to address a recently observed regression preventing some changes from merging | 21:09 | |
| opendevreview | Maksim Malchuk proposed openstack/kolla-ansible stable/yoga: Adding mariadb_port to wsrep sync status so alterative ports can be used https://review.opendev.org/c/openstack/kolla-ansible/+/901918 | 21:52 |
| opendevreview | Maksim Malchuk proposed openstack/kayobe stable/2023.1: kolla-bifrost: Add precheck for inspection allocation pool https://review.opendev.org/c/openstack/kayobe/+/902221 | 22:33 |
| opendevreview | Maksim Malchuk proposed openstack/kayobe stable/zed: kolla-bifrost: Add precheck for inspection allocation pool https://review.opendev.org/c/openstack/kayobe/+/902222 | 22:33 |
| opendevreview | Maksim Malchuk proposed openstack/kayobe stable/yoga: kolla-bifrost: Add precheck for inspection allocation pool https://review.opendev.org/c/openstack/kayobe/+/902223 | 22:33 |
| opendevreview | Maksim Malchuk proposed openstack/kayobe stable/2023.1: Honor custom ssh args in kayobe-ansible-user.yml https://review.opendev.org/c/openstack/kayobe/+/902224 | 22:37 |
| opendevreview | Maksim Malchuk proposed openstack/kayobe stable/zed: Honor custom ssh args in kayobe-ansible-user.yml https://review.opendev.org/c/openstack/kayobe/+/902225 | 22:38 |
| opendevreview | Maksim Malchuk proposed openstack/kayobe stable/yoga: Honor custom ssh args in kayobe-ansible-user.yml https://review.opendev.org/c/openstack/kayobe/+/902226 | 22:38 |
| opendevreview | Maksim Malchuk proposed openstack/kayobe stable/zed: docs: add information about enabling TLS to automated guide https://review.opendev.org/c/openstack/kayobe/+/902227 | 22:40 |
| opendevreview | Maksim Malchuk proposed openstack/kayobe stable/yoga: docs: add information about enabling TLS to automated guide https://review.opendev.org/c/openstack/kayobe/+/902228 | 22:41 |
| opendevreview | Verification of a change to openstack/kolla master failed: podman: install "rich" dependency https://review.opendev.org/c/openstack/kolla/+/902179 | 23:43 |
| opendevreview | Verification of a change to openstack/kolla master failed: fluentd: Add labels for transition to v5 https://review.opendev.org/c/openstack/kolla/+/901508 | 23:43 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!