| opendevreview | Michal Nasiadka proposed openstack/kolla stable/2023.1: Add comprehensive checks for container restarts https://review.opendev.org/c/openstack/kolla/+/892876 | 07:13 |
|---|---|---|
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/zed: Add comprehensive checks for container restarts https://review.opendev.org/c/openstack/kolla/+/892877 | 07:13 |
| opendevreview | Robin Klostermeyer proposed openstack/kolla-ansible master: Add ironic-prometheus-exporter https://review.opendev.org/c/openstack/kolla-ansible/+/874380 | 07:25 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/yoga: Add comprehensive checks for container restarts https://review.opendev.org/c/openstack/kolla/+/892878 | 08:21 |
| opendevreview | Matt Crees proposed openstack/kayobe master: CI: Migrate RabbitMQ queues during upgrade tasks https://review.opendev.org/c/openstack/kayobe/+/893137 | 08:39 |
| opendevreview | Matt Crees proposed openstack/kayobe master: CI: Migrate RabbitMQ queues during upgrade tasks https://review.opendev.org/c/openstack/kayobe/+/893137 | 08:42 |
| tony | Hello team | 09:45 |
| tony | I am trying to add two compute nodes to my openstack cluster wallaby but I am getting this error | 09:45 |
| tony | failed: [compute3] (item={'key': 'cron', 'value': {'container_name': 'cron', 'group': 'cron', 'enabled': True, 'image': 'kolla/ubuntu-source-cron:wallaby', 'environment': {'DUMMY_ENVIRONMENT': 'kolla_useless_env'}, 'volumes': ['/etc/kolla/cron/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/etc/timezone:/etc/timezone:ro', 'kolla_logs:/var/log/kolla/'], 'dimensions': {}}}) => {"ansible_loop_var": "item", "changed": true, "ite | 09:46 |
| tony | Any idea on how to go about it ? | 09:46 |
| mmalchuk | nope. provided information too small | 09:48 |
| tony | U mean more logs ? | 09:49 |
| tony | failed: [compute3] (item={'key': 'cron', 'value': {'container_name': 'cron', 'group': 'cron', 'enabled': True, 'image': 'kolla/ubuntu-source-cron:wallaby', 'environment': {'DUMMY_ENVIRONMENT': 'kolla_useless_env'}, 'volumes': ['/etc/kolla/cron/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/etc/timezone:/etc/timezone:ro', 'kolla_logs:/var/log/kolla/'], 'dimensions': {}}}) => {"ansible_loop_var": "item", "changed": true, "ite | 09:49 |
| mnasiadka | use paste.openstack.org and paste a bigger snippet | 09:50 |
| mmalchuk | tony also it would be better if you run ansible with -vvvv to get more info in these logs | 09:51 |
| tony | https://paste.openstack.org/show/bxaZSKaTtDKNjejUJbP8/ | 09:53 |
| tony | Ok | 09:53 |
| mmalchuk | folks, please merge trivial backports of documentation :https://review.opendev.org/q/I07ca5b82ecceda500e00260262c2a711c97b33bd | 09:54 |
| mmalchuk | and another one trivial fix: https://review.opendev.org/c/openstack/kolla-ansible/+/892837 | 09:55 |
| mmalchuk | thanks | 09:55 |
| opendevreview | Merged openstack/kolla stable/zed: Add comprehensive checks for container restarts https://review.opendev.org/c/openstack/kolla/+/892877 | 09:57 |
| tony | https://paste.openstack.org/show/bzC73Dh17jZmdcCNRbnw/ | 09:58 |
| tony | That is the detailed log | 09:59 |
| opendevreview | Merged openstack/kayobe master: Work around failing RabbitMQ precheck https://review.opendev.org/c/openstack/kayobe/+/893098 | 09:59 |
| mmalchuk | tony <compute3> Failed to connect to the host via ssh | 10:01 |
| mmalchuk | the issue in the 1st line) | 10:02 |
| mmalchuk | all other ansible stuff in the log not interested | 10:02 |
| mmalchuk | check your host/inventory/whatever related | 10:03 |
| tony | Which host would that be ? | 10:05 |
| opendevreview | Matt Crees proposed openstack/kayobe master: CI: Migrate RabbitMQ queues during upgrade tasks https://review.opendev.org/c/openstack/kayobe/+/893137 | 10:06 |
| tony | mmalchuk 127.0.0.1 localhost #10.10.13.32 controller3 10.10.13.27 controller1 10.10.13.28 controller2 10.10.13.29 compute1 10.10.13.30 compute2 10.10.34.182 deployment 10.10.13.38 compute3 10.10.13.37 compute4 | 10:06 |
| opendevreview | Matt Crees proposed openstack/kayobe master: CI: Migrate RabbitMQ queues during upgrade tasks https://review.opendev.org/c/openstack/kayobe/+/893137 | 10:06 |
| mmalchuk | tony check your inventory | 10:06 |
| opendevreview | Matt Crees proposed openstack/kayobe master: CI: Migrate RabbitMQ queues during upgrade tasks https://review.opendev.org/c/openstack/kayobe/+/893137 | 10:09 |
| tony | The inventory files look ok | 10:16 |
| mmalchuk | check your hosts from inventory | 10:28 |
| mmalchuk | at least compute3 as an error show | 10:29 |
| tony | (kolla-openstack) stack@deployment:~$ ansible -i multinode all -m ping [WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details localhost | SUCCESS => { "ansible_facts": { "discovered_interpreter_python": "/usr/bin/python3" }, "changed": false, "ping": "pong" } compute3 | SUCCESS => { "ansible_facts": { "discovered_interpreter_python": "/usr/bin/python3" }, "change | 10:33 |
| tony | I think if I had an issue that wouldnt have passed | 10:33 |
| mmalchuk | but an error show: <compute3> Failed to connect to the host via ssh | 10:35 |
| tony | I fully understand but I have checked all /etc/hosts files on all nodes and they are ok | 10:36 |
| mmalchuk | try compute3 instead of all and something other than ping module | 10:36 |
| SvenKieske | tony: check manually if this ssh works | 10:37 |
| mmalchuk | there may be problems if ~/.ssh/config exists and contain something not supported by ansible/kolla | 10:40 |
| tony | https://paste.openstack.org/show/bqdOuNWY78wAL8xm9Xf8/ | 10:41 |
| tony | That is the test I have done | 10:41 |
| tony | I am able to sssh everywhere | 10:41 |
| mmalchuk | you should check from the ansible host | 10:42 |
| mmalchuk | ssh to remote | 10:42 |
| mmalchuk | to compute3 | 10:42 |
| tony | (kolla-openstack) stack@deployment:~$ ssh compute3 Welcome to Ubuntu 20.04.6 LTS (GNU/Linux 5.4.0-159-generic x86_64) * Documentation: https://help.ubuntu.com * Management: https://landscape.canonical.com * Support: https://ubuntu.com/advantage System information as of Wed 30 Aug 2023 12:43:47 PM CAT System load: 0.21 Swap usage: 0% Users logged in: 0 Usage of /: 4.0% of 438.54GB Temperature: 57.0 C | 10:44 |
| tony | It works Ok | 10:44 |
| mmalchuk | ok. how did you got an error ? may be a new run after ssh check would pass? | 10:45 |
| mmalchuk | maybe issue in key fingerprint you are not accepted before? | 10:46 |
| mmalchuk | or whatever? | 10:46 |
| mmalchuk | as you said you add two compute nodes | 10:47 |
| mmalchuk | did you configure them the same way as previous nodes? | 10:48 |
| tony | I am regenerating the keys and check again | 10:52 |
| opendevreview | Merged openstack/kolla-ansible master: Add ironic-prometheus-exporter https://review.opendev.org/c/openstack/kolla-ansible/+/874380 | 10:56 |
| tony | They have also failed | 10:57 |
| mmalchuk | check the logs before, maybe you see next errors which caused by previous one | 11:07 |
| mmalchuk | anyway this is not an ansible playbook error but something with ssh as en error shows | 11:07 |
| opendevreview | Martin Hiner proposed openstack/kolla-ansible master: Add support of podman deployment https://review.opendev.org/c/openstack/kolla-ansible/+/799229 | 11:14 |
| SvenKieske | regex fans invited to take a look at: https://review.opendev.org/c/openstack/kolla-ansible/+/879426 ;) | 11:26 |
| kevko | Hi, folks , i see this error message on grafana dashboard (kolla-ansible deployed with custom home dashboard) https://github.com/grafana/grafana/issues/54574 ... | 11:36 |
| kevko | we should fix | 11:36 |
| kevko | but firstly ...do anyone know how to fix ? :) | 11:37 |
| mnasiadka | SvenKieske: I always loved those long regex strings in fluentd configs | 12:07 |
| SvenKieske | mnasiadka: well, they're still somewhat readable. I have seen worse :D | 12:23 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: ansible: bump min version to 2.14 and max to 2.15 https://review.opendev.org/c/openstack/kolla-ansible/+/887227 | 12:24 |
| mnasiadka | time to get back to merging this ;) | 12:24 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Change regex for the multiline parser at the fluentd rabbitmq input https://review.opendev.org/c/openstack/kolla-ansible/+/879426 | 12:25 |
| opendevreview | Matt Crees proposed openstack/kayobe master: CI: Migrate RabbitMQ queues during upgrade tasks https://review.opendev.org/c/openstack/kayobe/+/893137 | 12:25 |
| SvenKieske | I still think I spotted a bug there, but I might be wrong, it's hard to count this stuff manually :D | 12:26 |
| SvenKieske | kevko: what do you mean? this is an upstream issue no? you can deploy custom dashboards like described here[0] and they should work, no? | 12:27 |
| SvenKieske | kevko: [0]: https://grafana.com/docs/grafana/latest/administration/provisioning/#provision-folders-structure-from-filesystem-to-grafana | 12:28 |
| SvenKieske | I vaguely remember having done something like this in the past and it worked, but that might have been 2-3 years ago, so is there a bug in k-a now? | 12:28 |
| sahid | o? | 12:32 |
| SvenKieske | and a reminder to maybe look at the designate improvements, imho they are now in a rather good shape: https://review.opendev.org/c/openstack/kolla-ansible/+/878270 | 12:33 |
| sahid | o/ | 12:33 |
| sahid | I have question regarding kolla and requirements/upper-constraints.txt | 12:33 |
| sahid | how I can change from where this file is downloaded? | 12:33 |
| sahid | hum probably from here... | 12:35 |
| SvenKieske | o/ | 12:35 |
| SvenKieske | I guess the answer may depend on what you really want to achieve with that :/ :D | 12:35 |
| sahid | we would like to run without the need of internet | 12:36 |
| sahid | unfortunatly we are still on ussuri and that block kolla_toolbox_upper_constaints does not exist yet | 12:37 |
| SvenKieske | mhm, you could backport it, but I'm not sure how much work that would be | 12:42 |
| SvenKieske | anyone know's if Jason from the Let's Encrypt Change is still active in openstack? I see no activity since last september in openstack: https://review.opendev.org/c/openstack/kolla-ansible/+/741340 https://www.stackalytics.io/?user_id=jasonandersonatuchicago&project_type=openstack&release=all&metric=marks | 12:43 |
| SvenKieske | sahid: if you like offline install you might also be interested in this change: https://review.opendev.org/c/openstack/kolla/+/884368 | 12:45 |
| opendevreview | Bartosz Bezak proposed openstack/kolla-ansible stable/yoga: Added precheck for OpenSearch migration https://review.opendev.org/c/openstack/kolla-ansible/+/893172 | 12:46 |
| sahid | ack thank you SvenKieske | 12:47 |
| kevko | SvenKieske: no it's a bug of kolla-ansible ...or let's say there has to be a way how to do it correctly if user is using custome dashboard | 12:48 |
| SvenKieske | well the dashboard needs to be in place via git/docker/whatever before grafana can use it. IIRC I just altered the pipeline which produced our docker containers to include said dashboards, or did the git checkout in the grafana container, I don't remember it 100%. I still see no bug? | 12:49 |
| mnasiadka | mgoddard mnasiadka hrw bbezak frickler kevko SvenKieske mmalchuk gkoper jangutter - meeting in 10 | 12:50 |
| kevko | SvenKieske: kolla-ansible is copying grafana_custom_home_dashboard to /etc/kolla/grafana/... on target node and config.json is rendered so it is injected into /usr/share/grafana/public/dashboards/home.json in container ... so before grafana there is already injected custom home ..stil error | 12:52 |
| mnasiadka | raise a bug at least? :) | 12:52 |
| SvenKieske | ah, so you are saying it does not work? :D I was under the impression the linked grafana bug explains that it should work, but it's rather long and I just skimmed it | 12:53 |
| kevko | SvenKieske: what i want to say is that if kolla-ansible supports custom home dashboard out-of-the box ..user shouldn't do some additional steps to fix it | 12:53 |
| kevko | mnasiadka: I will, and i will also fix it :D | 12:53 |
| opendevreview | Bartosz Bezak proposed openstack/kolla-ansible stable/yoga: Added precheck for OpenSearch migration https://review.opendev.org/c/openstack/kolla-ansible/+/893172 | 12:53 |
| SvenKieske | thinking about it, isn't /usr/share/ the wrong directory for that? but it's really been some time since I looked at it, I need to check | 12:54 |
| opendevreview | Martin Hiner proposed openstack/ansible-collection-kolla master: Add Podman support https://review.opendev.org/c/openstack/ansible-collection-kolla/+/852240 | 12:54 |
| SvenKieske | kevko: okay, understood, I tend to agree :) | 12:54 |
| kevko | SvenKieske: https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana/#default_home_dashboard_path | 12:54 |
| kevko | SvenKieske: first find some information and then argument :D | 12:55 |
| SvenKieske | I just remember adding dashboards; I guess it wasn't even "home" dashboards, which seem kind of special cased | 12:55 |
| SvenKieske | a better path would still be /etc/grafana/ imho | 12:55 |
| kevko | SvenKieske: dashboards folder working as a charm in k-a | 12:56 |
| kevko | SvenKieske: if you will read bugreport i've send you will find that home dashboard is special case .. | 12:56 |
| kevko | SvenKieske: so you need to import again (the same home dashboard) via dashboard ... and error will dissapear | 12:57 |
| SvenKieske | okay, one thing that's standing out to me in the code is, that the home dashboard is owned by user root instead of grafana. but I don't know if that's on purpose, maybe grafana drops privs later? | 12:57 |
| kevko | SvenKieske: even if file is present in container | 12:57 |
| SvenKieske | that's weird | 12:57 |
| kevko | SvenKieske: no, it's about the database ... | 12:57 |
| kevko | SvenKieske: try yourself :D | 12:57 |
| kevko | SvenKieske: if i imported same dashboard ..it disappeared | 12:58 |
| kevko | SvenKieske: one more thing i need to check is just call grafana api to reload dashboards ...i think this will work ... if it will work ..i will change the restart logic to restart -> wait -> reload dashboards | 12:58 |
| kevko | that's it | 12:59 |
| mnasiadka | #startmeeting kolla | 13:00 |
| opendevmeet | Meeting started Wed Aug 30 13:00:04 2023 UTC and is due to finish in 60 minutes. The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot. | 13:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 13:00 |
| opendevmeet | The meeting name has been set to 'kolla' | 13:00 |
| mnasiadka | #topic rollcall | 13:00 |
| mnasiadka | o/ | 13:00 |
| mmalchuk | o/ | 13:00 |
| opendevreview | Bartosz Bezak proposed openstack/kolla-ansible stable/yoga: Remove incorrect variable from OpenSearch role https://review.opendev.org/c/openstack/kolla-ansible/+/893175 | 13:00 |
| mhiner | o/ | 13:01 |
| SvenKieske | o/ | 13:01 |
| bbezak | o/ | 13:01 |
| SvenKieske | it might be the case that frickler is busy with some router hardware, just pinged him privately | 13:03 |
| jsuazows | o/ | 13:03 |
| mnasiadka | #topic agenda | 13:03 |
| mnasiadka | * CI status | 13:03 |
| mnasiadka | * Release tasks | 13:03 |
| mnasiadka | * Regular stable releases (first meeting in a month) | 13:03 |
| mnasiadka | * Current cycle planning | 13:03 |
| mnasiadka | * Additional agenda (from whiteboard) | 13:03 |
| mnasiadka | * Open discussion | 13:03 |
| mnasiadka | #topic CI status | 13:03 |
| mnasiadka | overall green I think, especially master | 13:03 |
| mnasiadka | if there's anything broken - please update the whiteboard | 13:03 |
| mnasiadka | #topic Release tasks | 13:04 |
| mnasiadka | it's cycle highlights week | 13:04 |
| mnasiadka | I'll try to compose something and will add let's encrypt and podman, because we should get it in | 13:04 |
| SvenKieske | the LE change author seems pretty unresponsive since last september, see my message above. maybe someone could adopt the change? I guess it's only minor stuff needed | 13:05 |
| kevko | i will work on comments .. | 13:05 |
| mnasiadka | SvenKieske: kevko is now updating it | 13:05 |
| SvenKieske | nice :) | 13:05 |
| mnasiadka | ok, it's R-5 and we have one task | 13:05 |
| mnasiadka | [all] Check for new versions of infrastructure components | 13:05 |
| mnasiadka | #link https://docs.openstack.org/kolla/latest/contributor/release-management.html#r-5-cycle-highlights-deadline | 13:05 |
| mnasiadka | anybody happy to pick this up? | 13:06 |
| kevko | SvenKieske: did you diff between last patchsetset from jason and my last patchset ? :D | 13:06 |
| SvenKieske | ok, time for dumb questions: what needs to be done? :D | 13:06 |
| kevko | SvenKieske: there is massive change which is tested | 13:06 |
| kevko | before it was not working properly | 13:06 |
| SvenKieske | ah the link explains it | 13:06 |
| mnasiadka | SvenKieske: there's a list of components listed, basically skim kolla docker images for versions and check if there are newer | 13:07 |
| mnasiadka | I mean scan | 13:07 |
| SvenKieske | I feel like that list is incomplete, no? if I should just check upstream I can. if this needs to be checked with supported distros etc. it's much more work. | 13:08 |
| SvenKieske | is this just about upstream checking (reporting to..where?) I'm okay doing that. | 13:08 |
| SvenKieske | help appreciated, as I notice this also includes all prometheus exporters :D | 13:09 |
| mnasiadka | yeah, we should update that list | 13:09 |
| mnasiadka | I think let's check only for versions that we control - as in prometheus exporters and similar things | 13:09 |
| SvenKieske | so everything where we pull directly from upstream (repos)? still not clear to me, sorry. | 13:10 |
| mnasiadka | yes | 13:10 |
| SvenKieske | okay, understood, seems reasonable. :) | 13:10 |
| mnasiadka | so in some cases we take a repo with some major version - like in rmq/mariadb case | 13:10 |
| mnasiadka | I think we bumped RMQ and MariaDB already, so it's off the list | 13:11 |
| mnasiadka | ceph is rather dependent on what we get from debian/ubuntu/centos repos | 13:11 |
| mnasiadka | ok, if you need any help - shout out :) | 13:11 |
| SvenKieske | cool | 13:11 |
| mnasiadka | #topic Current cycle planning | 13:12 |
| mnasiadka | So, Let's Encrypt - kevko will update based on comments | 13:12 |
| mnasiadka | Podman - mhiner need any help, or we just need to start reviewing? | 13:12 |
| mhiner | Yes, I need help. Can you look into whiteboard for tldr? | 13:13 |
| mhiner | What I don't understand is why this issue only happens on aarch64 debian test case | 13:13 |
| mnasiadka | K-A Podman patch https://review.opendev.org/c/openstack/kolla-ansible/+/799229 failed test: https://zuul.opendev.org/t/openstack/build/781e142dcb194767a130ed3670226aa5/logs | 13:14 |
| mnasiadka | Seems like the ansible insde of kolla_toolbox is unable to write in it's log file just for this single test case. Why could that be? | 13:14 |
| mnasiadka | kolla-ansible/ansible/roles/common/handlers/main.yml:2 fails with: [WARNING]: log file at /var/log/kolla/ansible.log is not writeable and we cannot create it, aborting | 13:14 |
| mnasiadka | aarch64 only? huh | 13:14 |
| SvenKieske | are we sure that the kolla_toolbox container has cap_audit? because that's a common way to not be able to write. I know the patchset enables this cap, but maybe not globally enough? just a random thought. | 13:15 |
| opendevreview | Bartosz Bezak proposed openstack/kolla-ansible stable/yoga: Added precheck for OpenSearch migration https://review.opendev.org/c/openstack/kolla-ansible/+/893172 | 13:15 |
| mnasiadka | mhiner: https://63942ee63c42f3e33db1-2a052569d3facbabacb331e67cb0b17e.ssl.cf2.rackcdn.com/799229/84/check-arm64/kolla-ansible-debian-aarch64-podman/781e142/primary/logs/container_logs/kolla_toolbox.txt - that looks a bit weird? | 13:15 |
| kevko | zun also failing for debian | 13:15 |
| SvenKieske | mhiner: can you maybe add a simple "echo 'test' >> /var/log/kolla/ansible.log" somewhere to rule out bugs in ansible/python? | 13:16 |
| SvenKieske | and maybe also an echo "test" > /tmp/testfile.txt or something to rule out permission problems with /var/log/ | 13:17 |
| mhiner | to be clear, inside of the kolla_toolbox right? | 13:17 |
| SvenKieske | unless someone has a better idea to narrow it down. I didn't take a closer look. | 13:17 |
| mhiner | and iirc, kolla_toolbox is in privileged mode so it should not need any additional caps | 13:18 |
| SvenKieske | mhiner: sure, inside the container, with the same user | 13:18 |
| SvenKieske | mhiner: right, if it's privileged it should suffice. but weird that it's only failing on aarch64 like mnasiadka mentioned. | 13:18 |
| mnasiadka | well, basically as you can see we have no clue, but keep debugging :) | 13:19 |
| mnasiadka | #topic Additional agenda (from whiteboard) | 13:20 |
| mnasiadka | One was the podman one we discussed | 13:20 |
| SvenKieske | we could do one recheck to rule out spurious sunflare errors in the DC :) | 13:20 |
| mmalchuk | have opend discussion related to podman | 13:20 |
| mnasiadka | ok, frickler - octavia jobboard - I think it's getting reviews | 13:21 |
| mnasiadka | SvenKieske: asking for review/feedback on https://review.opendev.org/c/openstack/kolla-ansible/+/874769 | 13:21 |
| mnasiadka | I think we did speak about it last time, or outside of a meeting | 13:22 |
| mmalchuk | commented octavia jobboard | 13:22 |
| frickler | ack, thx mmalchuk | 13:22 |
| SvenKieske | I | 13:22 |
| mnasiadka | if the author would add some CI tests and move the python inline code to a normal file - would be nice | 13:22 |
| SvenKieske | I'll try to poke the author of said patch | 13:22 |
| SvenKieske | I forgot to remove it from the whiteboard | 13:22 |
| mnasiadka | SvenKieske: the kolla patch you wanted to talk about is already merged - so I guess that concludes :) | 13:23 |
| mnasiadka | k.yarovoy: podman for kolla - https://review.opendev.org/c/openstack/kolla/+/825791/10 one patchset left, but it will need some efforts on rebase and adaptation of ci. Is there any planned way to approach it? | 13:23 |
| SvenKieske | yeah, didn't update the whiteboard, just did it, sorry. | 13:23 |
| mnasiadka | ok then, would be nice to have support for podman build in Kolla | 13:24 |
| kevko | btw, all container logs are broken https://63942ee63c42f3e33db1-2a052569d3facbabacb331e67cb0b17e.ssl.cf2.rackcdn.com/799229/84/check-arm64/kolla-ansible-debian-aarch64-podman/781e142/primary/logs/container_logs/index.html | 13:24 |
| mnasiadka | mhiner: k.yarovoy is your colleague? | 13:24 |
| opendevreview | Matt Crees proposed openstack/kayobe master: CI: Migrate RabbitMQ queues during upgrade tasks https://review.opendev.org/c/openstack/kayobe/+/893137 | 13:24 |
| mhiner | mnasiadka: yes | 13:24 |
| mnasiadka | mhiner: if he could rebase the patch (now it's in merge conflict) - I can help with CI | 13:24 |
| opendevreview | Matt Crees proposed openstack/kayobe master: CI: Migrate RabbitMQ queues during upgrade tasks https://review.opendev.org/c/openstack/kayobe/+/893137 | 13:25 |
| mnasiadka | mhiner: he's not here, so maybe you could convey the message :) | 13:25 |
| mhiner | also, he did not write it bet he wanted to ask if someone could pick it up bcs he won't be able to attend it for at least two next weeks | 13:25 |
| mnasiadka | ah ok | 13:25 |
| mnasiadka | I'll try rebasing on my own then and adding CI | 13:25 |
| mhiner | thank you | 13:25 |
| SvenKieske | kevko: can you open a bug? I'm not sure what changed recently in the logging infrastructure | 13:26 |
| SvenKieske | sample error msg: Error: invalid argument "all" for "--tail" flag: strconv.ParseInt: parsing "all": invalid syntax | 13:26 |
| mnasiadka | SvenKieske: it's the podman patch - probably logging is broken only there | 13:26 |
| mnasiadka | next one: | 13:26 |
| mnasiadka | jsuazows - Easy bug fixes proposed: | 13:26 |
| mnasiadka | OVS_DPDK task name fix | 13:26 |
| mnasiadka | https://review.opendev.org/c/openstack/kolla-ansible/+/892613 | 13:26 |
| mnasiadka | Horizon SSO fix | 13:26 |
| mnasiadka | https://review.opendev.org/c/openstack/kolla-ansible/+/891923 | 13:26 |
| SvenKieske | the horizon patch just needs people with these weird +2 powers ;) | 13:27 |
| mnasiadka | yeah, let's hope it doesn't break anything :) | 13:27 |
| mmalchuk | mnasiadka another one trivial: https://review.opendev.org/c/openstack/kolla-ansible/+/892837 | 13:27 |
| SvenKieske | I'll look at dpdk even if I have no real interest in it. | 13:27 |
| mnasiadka | but I guess I did override that somewhere locally for SSO to work | 13:27 |
| mnasiadka | dpdk is trivial fix | 13:28 |
| SvenKieske | the SSO stuff was rather good tested, afaik. don't know if the github issue is linked somewhere | 13:28 |
| mnasiadka | bbezak: willing to have a look in the SSO patch? ^^ | 13:28 |
| kevko | mnasiadka: dpdk approved | 13:30 |
| mmalchuk | and server-status and kayobe patches from the last meeting question?) | 13:30 |
| SvenKieske | nvm, I thought about another SSO bug, rather that one was about OIDC.. | 13:31 |
| mnasiadka | ok, those are done | 13:31 |
| mnasiadka | mmalchuk: can you post a link so other cores can have a look? | 13:31 |
| mmalchuk | #link https://review.opendev.org/c/openstack/kayobe/+/861397 | 13:32 |
| mmalchuk | #link https://review.opendev.org/c/openstack/kayobe/+/879554 | 13:32 |
| opendevreview | Jan Gutter proposed openstack/kolla-ansible master: etcd: update to v3.4 https://review.opendev.org/c/openstack/kolla-ansible/+/890465 | 13:32 |
| mmalchuk | #link https://review.opendev.org/c/openstack/kolla/+/891098 | 13:32 |
| mnasiadka | ok then | 13:34 |
| kevko | https://review.opendev.org/q/owner:lukas.mrtvy%2540gmail.com << another sso fixes abandoned ... guesswhat8 abandoned | 13:34 |
| mnasiadka | #topic Open discussion | 13:34 |
| mmalchuk | 1. "bad practice" | 13:34 |
| mmalchuk | https://review.opendev.org/c/openstack/ansible-collection-kolla/+/892990 | 13:34 |
| mmalchuk | unset or comment? | 13:34 |
| mnasiadka | I have no preference, but it seems like a cosmetic change with no gain | 13:37 |
| mmalchuk | imho default+omit is a good practice | 13:37 |
| mnasiadka | as long as we don't have coding guidelines with best practices documented and align all code to this - I don't think it makes any sense | 13:37 |
| frickler | gain of good style, I'm +2 and we can add that to coding guidelines? | 13:38 |
| SvenKieske | {{ virtualenv is none | ternary(omit, virtualenv) }}" is way harder to read than: {{ virtualenv | default(omit) }} | 13:38 |
| mnasiadka | do we have any coding guidelines? :) | 13:38 |
| frickler | didn't we link to that last week? | 13:39 |
| mnasiadka | well, ternary is not there today, it's only afaik in podman patch | 13:39 |
| SvenKieske | we could invent some :) I'm all for it, because some reviews boil down to taste. I think a guideline can help cut discussions short | 13:39 |
| SvenKieske | it's the next best thing to: always do what the linter tells you | 13:39 |
| mmalchuk | anyone have a link to coding guidelines? | 13:39 |
| mnasiadka | first of all, that's not the only occurence of using "is not none" for an empty defined var in ansible-collection-kolla | 13:40 |
| mnasiadka | so either we change all of them or none | 13:40 |
| frickler | https://docs.openstack.org/kolla/latest/contributor/kolla-design-philosophy.html , maybe need to extend the domain from design to coding? | 13:40 |
| SvenKieske | like the python "black" approach, or gofmt, or rustfmt | 13:40 |
| mmalchuk | +1 for all of them) | 13:40 |
| mnasiadka | second of all, maybe newer ansible-lint at least warns on something like that, then it would be easier to enforce | 13:40 |
| mnasiadka | or we need to write a custom rule | 13:40 |
| SvenKieske | so let me rephrase: I don't like coding guidelines, I like syntax enforced by a linter (basically coding guidelines automated as code). If I can't have that; I like guidelines. | 13:41 |
| mmalchuk | ansible-lint quiet about this | 13:41 |
| mnasiadka | we use some old version | 13:41 |
| mnasiadka | maybe it's time to bump up | 13:41 |
| mmalchuk | may be never version? | 13:41 |
| mnasiadka | and then add some custom rules if needed | 13:41 |
| SvenKieske | ansible-lint is really a pain because they, at least sometimes, change quite a lot of stuff which results in new errors and warnings | 13:42 |
| frickler | maybe that cannot be caught by a linter, because it needs too much semantic understanding | 13:42 |
| SvenKieske | mnasiadka: put ansible-lint on the "to check for updates" list? ;) | 13:42 |
| mnasiadka | SvenKieske: might be, but that will be painful | 13:42 |
| mmalchuk | so lets merge https://review.opendev.org/c/openstack/ansible-collection-kolla/+/892990 and fix all other places later with lint | 13:43 |
| mnasiadka | ok, anyway - I think linter that enforces it is the way | 13:43 |
| mnasiadka | mmalchuk: I disagree | 13:43 |
| mmalchuk | why? | 13:43 |
| mmalchuk | starting from coding guidelines and lint ? | 13:44 |
| mnasiadka | I agree with SvenKieske that writing coding guidelines is a waste of resources, it's better to do proper linting. | 13:45 |
| mnasiadka | I'm fine with fixing bad practices, but we need to check for them and fix them everywhere, not in just one role. | 13:45 |
| mmalchuk | upgrade lint and make job non voting? | 13:45 |
| mnasiadka | upgrade lint and stack lint update on top of patches that make us compliant | 13:45 |
| SvenKieske | I don't know if a tool can enforce good style on playbooks, the ansible-lint stuff is okayish, but had some drawbacks in the past. as always: a linter or coding guidelines will incur additional work | 13:46 |
| SvenKieske | but it hopefully will make stuff more consistent and maybe also catch some errors | 13:46 |
| mmalchuk | do we really have linter like black ? | 13:46 |
| mmalchuk | which changes the code | 13:46 |
| SvenKieske | ansible-lint --write . | 13:47 |
| SvenKieske | I don't know how reliable that is, but it exists :) | 13:47 |
| mmalchuk | wow... thanks. didn't know | 13:48 |
| mmalchuk | will check all the code))) | 13:48 |
| mnasiadka | ok then | 13:50 |
| SvenKieske | as long as we realize that everything in live has also drawbacks it might be a worthwhile extension :) | 13:50 |
| mnasiadka | anything else? | 13:51 |
| jangutter | fans of ARA logging | https://review.opendev.org/c/openstack/kolla-ansible/+/891097 :-p | 13:51 |
| mmalchuk | lack of kayobe cores | 13:51 |
| mnasiadka | jangutter: I'm still not convinced around the additional time of jobs | 13:52 |
| SvenKieske | you can scratch kayobe from that sentence ;) | 13:52 |
| mmalchuk | :) | 13:52 |
| mmalchuk | anyway | 13:52 |
| SvenKieske | regarding ARA: It might be nice to be able to enable it on a case-by-case basis? | 13:52 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Add support of podman deployment https://review.opendev.org/c/openstack/kolla-ansible/+/799229 | 13:53 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Fix podman logs https://review.opendev.org/c/openstack/kolla-ansible/+/893187 | 13:53 |
| jangutter | mnasiadka : that is indeed a fair reason not to do it. I can easily change it so that it can be triggered by something in the commit message. | 13:53 |
| tony | Hello Team, .. I am facing the issue regarding installing new compute nodes on my wallaby openstack cluster using kolla-ansible, I am getting the error below | 13:54 |
| mnasiadka | we might go that path, ara might be sometimes useful | 13:54 |
| mnasiadka | tony: wait a moment, there's a meeting in progress | 13:54 |
| mnasiadka | jangutter: can you rework that please? | 13:54 |
| tony | Oooh My Bad | 13:54 |
| mmalchuk | btw hell of ansible-lint in bifrost: https://review.opendev.org/q/project:openstack/bifrost+topic:ansible-lint | 13:54 |
| jangutter | mnasiadka: ack, won't take too long - we happy with the interface where "#log_ara" or something in the commit message triggers it? | 13:55 |
| mnasiadka | jangutter: I think so | 13:55 |
| mnasiadka | jangutter: we could add docs around how to use it somewhere in Kolla-Ansible contributor guide? | 13:55 |
| jangutter | mnasiadka: good idea, I'll add it there! | 13:56 |
| mnasiadka | thanks! | 13:56 |
| mnasiadka | ok, I guess it's enough for today :) | 13:56 |
| mnasiadka | Thanks all for coming! | 13:56 |
| mnasiadka | #endmeeting | 13:56 |
| opendevmeet | Meeting ended Wed Aug 30 13:56:27 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 13:56 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/kolla/2023/kolla.2023-08-30-13.00.html | 13:56 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/kolla/2023/kolla.2023-08-30-13.00.txt | 13:56 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/kolla/2023/kolla.2023-08-30-13.00.log.html | 13:56 |
| SvenKieske | thanks for all your efforts o/ | 13:58 |
| tony | @mnasiadka, Would now be a good time to share my findings ? | 14:00 |
| mnasiadka | tony: yes, but for pasting logs/code/whatever please use paste.openstack.org | 14:00 |
| tony | Sure | 14:01 |
| tony | So I can confirm that the ssh is not the issue as for other services / containers the ssh works minus for toolbox, fluentd and cron | 14:02 |
| tony | i also found out that on the new compute nodes, they have docker version 24 after doing the bootstrap but the older ones have 20. | 14:02 |
| kevko | tony: probably because state=present in ansible ? :D | 14:04 |
| tony | @kevko, I didnt manipulate anything in ansible | 14:05 |
| tony | But my question is... Could that be the reason for the failure ? | 14:05 |
| mmalchuk | lost my internet connection | 14:05 |
| mmalchuk | mnasiadka thanks | 14:06 |
| opendevreview | Bartosz Bezak proposed openstack/kolla-ansible stable/yoga: Remove incorrect variable from OpenSearch role https://review.opendev.org/c/openstack/kolla-ansible/+/893175 | 14:09 |
| opendevreview | Merged openstack/kayobe master: Speed up calls to Bifrost https://review.opendev.org/c/openstack/kayobe/+/882951 | 14:11 |
| kevko | tony where is your log or something ? | 14:13 |
| tony | Let me share | 14:14 |
| kevko | tony use paste.openstack.org and share the link | 14:14 |
| opendevreview | Bartosz Bezak proposed openstack/kolla-ansible stable/yoga: Added precheck for OpenSearch migration https://review.opendev.org/c/openstack/kolla-ansible/+/893172 | 14:15 |
| tony | Here is the link | 14:15 |
| tony | https://paste.openstack.org/show/bNxHwZb4yqmTFUX7ulX6/ | 14:15 |
| opendevreview | Bartosz Bezak proposed openstack/kolla-ansible stable/yoga: Added precheck for OpenSearch migration https://review.opendev.org/c/openstack/kolla-ansible/+/893172 | 14:16 |
| kevko | tony: incompatibility between docker and kolla module | 14:16 |
| kevko | tony: you need to backport one commit to your downstream kolla-ansible repository | 14:16 |
| kevko | tony: or amend your kolla-ansible code on disk | 14:17 |
| kevko | let me check | 14:17 |
| kevko | tony: do you have this change applied ? https://review.opendev.org/c/openstack/kolla-ansible/+/872671 | 14:18 |
| tony | Let me check | 14:19 |
| tony | @kevko, Am sorry but I am abit on the change and unaware on how to change it, Any guidance will be appreciated | 14:22 |
| kevko | grep KernelMemory your_path_for_kolla_ansible_code/ansible/library/kolla_docker.py | 14:23 |
| kevko | tony: ^^ | 14:23 |
| tony | on it | 14:24 |
| kevko | if grep -rq self._dimensions_kernel_memory_removed your_path_for_kolla_ansible_code/; then echo "commit included"; else echo "commit not included";fi | 14:25 |
| tony | results of the grep KernelMemory your_path_for_kolla_ansible_code/ansible/library/kolla_docker.py | 14:29 |
| tony | https://paste.openstack.org/show/bDVCZYLpkXbzEEnfdHTz/ | 14:29 |
| kevko | tony: well, you have old version of kolla-ansible code | 14:30 |
| kevko | tony: just update your kolla-ansible code and it will work | 14:30 |
| tony | Would this command pip update ./kolla-ansible be enough ? | 14:33 |
| opendevreview | Pierre Riteau proposed openstack/kayobe stable/2023.1: Speed up calls to Bifrost https://review.opendev.org/c/openstack/kayobe/+/893201 | 14:33 |
| opendevreview | Pierre Riteau proposed openstack/kayobe stable/zed: Speed up calls to Bifrost https://review.opendev.org/c/openstack/kayobe/+/893202 | 14:33 |
| opendevreview | Pierre Riteau proposed openstack/kayobe stable/yoga: Speed up calls to Bifrost https://review.opendev.org/c/openstack/kayobe/+/893203 | 14:33 |
| opendevreview | Pierre Riteau proposed openstack/kayobe stable/xena: Speed up calls to Bifrost https://review.opendev.org/c/openstack/kayobe/+/893204 | 14:33 |
| opendevreview | Pierre Riteau proposed openstack/kayobe stable/wallaby: Speed up calls to Bifrost https://review.opendev.org/c/openstack/kayobe/+/893205 | 14:34 |
| opendevreview | Merged openstack/kolla-ansible master: Fixes task name in notify module to the actual task name https://review.opendev.org/c/openstack/kolla-ansible/+/892613 | 14:49 |
| opendevreview | Merged openstack/kolla-ansible master: Drop useless DUMMY_ENVIRONMENT environment for cron container https://review.opendev.org/c/openstack/kolla-ansible/+/892837 | 14:49 |
| opendevreview | Merged openstack/kolla-ansible master: Fixes WEBSSO_KEYSTONE_URL Value https://review.opendev.org/c/openstack/kolla-ansible/+/891923 | 14:50 |
| opendevreview | Verification of a change to openstack/kolla-ansible stable/yoga failed: opensearch: alter path after using rpm/deb packaging https://review.opendev.org/c/openstack/kolla-ansible/+/884376 | 14:52 |
| opendevreview | Maksim Malchuk proposed openstack/kolla-ansible stable/2023.1: Fixes task name in notify module to the actual task name https://review.opendev.org/c/openstack/kolla-ansible/+/893206 | 14:59 |
| opendevreview | Maksim Malchuk proposed openstack/kolla-ansible stable/zed: Fixes task name in notify module to the actual task name https://review.opendev.org/c/openstack/kolla-ansible/+/893207 | 14:59 |
| opendevreview | Maksim Malchuk proposed openstack/kolla-ansible stable/2023.1: Drop useless DUMMY_ENVIRONMENT environment for cron container https://review.opendev.org/c/openstack/kolla-ansible/+/893208 | 15:02 |
| opendevreview | Maksim Malchuk proposed openstack/kolla-ansible stable/zed: Drop useless DUMMY_ENVIRONMENT environment for cron container https://review.opendev.org/c/openstack/kolla-ansible/+/893209 | 15:02 |
| opendevreview | Maksim Malchuk proposed openstack/kolla-ansible stable/yoga: Drop useless DUMMY_ENVIRONMENT environment for cron container https://review.opendev.org/c/openstack/kolla-ansible/+/893210 | 15:02 |
| kevko | tony: no | 15:05 |
| kevko | tony: for wallaby install wallaby version from git | 15:05 |
| kevko | from stable branch | 15:05 |
| alistarle | hello guy's, we are lacking a second +2 on our fixup patch for deploying ironic in kolla, if you think you can merge it :) https://review.opendev.org/c/openstack/kolla-ansible/+/886747 | 15:06 |
| opendevreview | Maksim Malchuk proposed openstack/kolla-ansible stable/2023.1: Fixes WEBSSO_KEYSTONE_URL Value https://review.opendev.org/c/openstack/kolla-ansible/+/893212 | 15:06 |
| opendevreview | Maksim Malchuk proposed openstack/kolla-ansible stable/zed: Fixes WEBSSO_KEYSTONE_URL Value https://review.opendev.org/c/openstack/kolla-ansible/+/893213 | 15:07 |
| opendevreview | Maksim Malchuk proposed openstack/kolla-ansible stable/2023.1: Fix D001 Line too long in mariadb-guide https://review.opendev.org/c/openstack/kolla-ansible/+/893214 | 15:15 |
| opendevreview | Maksim Malchuk proposed openstack/kolla-ansible stable/zed: Fix D001 Line too long in mariadb-guide https://review.opendev.org/c/openstack/kolla-ansible/+/893215 | 15:16 |
| SvenKieske | mnasiadka: imho this needs relnotes as suddenly ironic-tftp is not working on all IPs anymore? https://review.opendev.org/c/openstack/kolla-ansible/+/886747 | 15:16 |
| opendevreview | Maksim Malchuk proposed openstack/kolla-ansible stable/yoga: Fix D001 Line too long in mariadb-guide https://review.opendev.org/c/openstack/kolla-ansible/+/893216 | 15:16 |
| opendevreview | Maksim Malchuk proposed openstack/kolla-ansible stable/xena: Fix D001 Line too long in mariadb-guide https://review.opendev.org/c/openstack/kolla-ansible/+/893217 | 15:16 |
| SvenKieske | I replied in the (merged) patch discussion, sometimes we are too fast imho, when it comes to merging: https://review.opendev.org/c/openstack/kolla-ansible/+/886747 | 15:39 |
| SvenKieske | to be fair, the patchset is not that new, but I didn't notice it until now. my bad I guess. | 15:40 |
| kevko | drop release notes :D | 15:45 |
| tony | Hey Kevko | 15:54 |
| tony | The upgrade helped | 15:54 |
| tony | Thanks a lot | 15:54 |
| SvenKieske | kevko: I was bitten enough times by missing relnotes for crucial stuff just changing without any notice in openstack. I think we should and can do better. or just give up and publish a git log for relnotes | 16:05 |
| SvenKieske | but we had that discussion already some weeks ago. | 16:06 |
| SvenKieske | https://keepachangelog.com | 16:06 |
| SvenKieske | this is basically https://semver.org/ for changelogs/release notes. | 16:07 |
| opendevreview | Merged openstack/kolla master: Add server-status handler to Rocky/Centos Apache conf https://review.opendev.org/c/openstack/kolla/+/891098 | 16:08 |
| opendevreview | Maksim Malchuk proposed openstack/kolla stable/2023.1: Add server-status handler to Rocky/Centos Apache conf https://review.opendev.org/c/openstack/kolla/+/893220 | 16:09 |
| opendevreview | Maksim Malchuk proposed openstack/kolla stable/zed: Add server-status handler to Rocky/Centos Apache conf https://review.opendev.org/c/openstack/kolla/+/893241 | 16:09 |
| opendevreview | Maksim Malchuk proposed openstack/kolla stable/yoga: Add server-status handler to Rocky/Centos Apache conf https://review.opendev.org/c/openstack/kolla/+/893242 | 16:09 |
| opendevreview | Maksim Malchuk proposed openstack/kolla stable/xena: Add server-status handler to Rocky/Centos Apache conf https://review.opendev.org/c/openstack/kolla/+/893243 | 16:09 |
| opendevreview | Felipe Figueroa Vergara proposed openstack/kolla master: Adds TAAS Neutron plugin to support OVS port mirrors https://review.opendev.org/c/openstack/kolla/+/885151 | 16:14 |
| *** ralonsoh is now known as ralonsoh_ooo | 16:20 | |
| opendevreview | Verification of a change to openstack/kolla-ansible stable/yoga failed: opensearch: alter path after using rpm/deb packaging https://review.opendev.org/c/openstack/kolla-ansible/+/884376 | 16:24 |
| opendevreview | Merged openstack/kolla-ansible master: Use better default bind address for ironic-tftp https://review.opendev.org/c/openstack/kolla-ansible/+/886747 | 17:03 |
| opendevreview | Hao Wang proposed openstack/kolla-ansible master: Support deploy Zaqar with kolla-ansible https://review.opendev.org/c/openstack/kolla-ansible/+/892615 | 17:20 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!