Wednesday, 2023-08-30

opendevreviewMichal Nasiadka proposed openstack/kolla stable/2023.1: Add comprehensive checks for container restarts  https://review.opendev.org/c/openstack/kolla/+/89287607:13
opendevreviewMichal Nasiadka proposed openstack/kolla stable/zed: Add comprehensive checks for container restarts  https://review.opendev.org/c/openstack/kolla/+/89287707:13
opendevreviewRobin Klostermeyer proposed openstack/kolla-ansible master: Add ironic-prometheus-exporter  https://review.opendev.org/c/openstack/kolla-ansible/+/87438007:25
opendevreviewMichal Nasiadka proposed openstack/kolla stable/yoga: Add comprehensive checks for container restarts  https://review.opendev.org/c/openstack/kolla/+/89287808:21
opendevreviewMatt Crees proposed openstack/kayobe master: CI: Migrate RabbitMQ queues during upgrade tasks  https://review.opendev.org/c/openstack/kayobe/+/89313708:39
opendevreviewMatt Crees proposed openstack/kayobe master: CI: Migrate RabbitMQ queues during upgrade tasks  https://review.opendev.org/c/openstack/kayobe/+/89313708:42
tonyHello team 09:45
tonyI am trying to add two compute nodes to my openstack cluster wallaby but I am getting this error09:45
tonyfailed: [compute3] (item={'key': 'cron', 'value': {'container_name': 'cron', 'group': 'cron', 'enabled': True, 'image': 'kolla/ubuntu-source-cron:wallaby', 'environment': {'DUMMY_ENVIRONMENT': 'kolla_useless_env'}, 'volumes': ['/etc/kolla/cron/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/etc/timezone:/etc/timezone:ro', 'kolla_logs:/var/log/kolla/'], 'dimensions': {}}}) => {"ansible_loop_var": "item", "changed": true, "ite09:46
tonyAny idea on how to go about it ?09:46
mmalchuknope. provided information too small09:48
tonyU mean more logs ?09:49
tonyfailed: [compute3] (item={'key': 'cron', 'value': {'container_name': 'cron', 'group': 'cron', 'enabled': True, 'image': 'kolla/ubuntu-source-cron:wallaby', 'environment': {'DUMMY_ENVIRONMENT': 'kolla_useless_env'}, 'volumes': ['/etc/kolla/cron/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/etc/timezone:/etc/timezone:ro', 'kolla_logs:/var/log/kolla/'], 'dimensions': {}}}) => {"ansible_loop_var": "item", "changed": true, "ite09:49
mnasiadkause paste.openstack.org and paste a bigger snippet09:50
mmalchuktony also it would be better if you run ansible with -vvvv to get more info in these logs09:51
tonyhttps://paste.openstack.org/show/bxaZSKaTtDKNjejUJbP8/09:53
tonyOk09:53
mmalchukfolks, please merge trivial backports of documentation :https://review.opendev.org/q/I07ca5b82ecceda500e00260262c2a711c97b33bd09:54
mmalchukand another one trivial fix: https://review.opendev.org/c/openstack/kolla-ansible/+/89283709:55
mmalchukthanks09:55
opendevreviewMerged openstack/kolla stable/zed: Add comprehensive checks for container restarts  https://review.opendev.org/c/openstack/kolla/+/89287709:57
tonyhttps://paste.openstack.org/show/bzC73Dh17jZmdcCNRbnw/09:58
tonyThat is the detailed log09:59
opendevreviewMerged openstack/kayobe master: Work around failing RabbitMQ precheck  https://review.opendev.org/c/openstack/kayobe/+/89309809:59
mmalchuktony <compute3> Failed to connect to the host via ssh10:01
mmalchukthe issue in the 1st line)10:02
mmalchukall other ansible stuff in the log not interested10:02
mmalchukcheck your host/inventory/whatever related10:03
tonyWhich host would that be ?10:05
opendevreviewMatt Crees proposed openstack/kayobe master: CI: Migrate RabbitMQ queues during upgrade tasks  https://review.opendev.org/c/openstack/kayobe/+/89313710:06
tonymmalchuk 127.0.0.1 localhost #10.10.13.32 controller3 10.10.13.27 controller1 10.10.13.28 controller2 10.10.13.29 compute1 10.10.13.30 compute2 10.10.34.182 deployment 10.10.13.38 compute3 10.10.13.37 compute410:06
opendevreviewMatt Crees proposed openstack/kayobe master: CI: Migrate RabbitMQ queues during upgrade tasks  https://review.opendev.org/c/openstack/kayobe/+/89313710:06
mmalchuktony check your inventory10:06
opendevreviewMatt Crees proposed openstack/kayobe master: CI: Migrate RabbitMQ queues during upgrade tasks  https://review.opendev.org/c/openstack/kayobe/+/89313710:09
tonyThe inventory files look ok 10:16
mmalchukcheck your hosts from inventory10:28
mmalchukat least compute3 as an error show10:29
tony(kolla-openstack) stack@deployment:~$ ansible -i multinode all -m ping [WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details localhost | SUCCESS => {     "ansible_facts": {         "discovered_interpreter_python": "/usr/bin/python3"     },     "changed": false,     "ping": "pong" } compute3 | SUCCESS => {     "ansible_facts": {         "discovered_interpreter_python": "/usr/bin/python3"     },     "change10:33
tonyI think if I had an issue that wouldnt have passed 10:33
mmalchukbut an error show: <compute3> Failed to connect to the host via ssh10:35
tonyI fully understand but I have checked all  /etc/hosts files on all nodes and they are ok 10:36
mmalchuktry compute3 instead of all and something other than ping module10:36
SvenKiesketony: check manually if this ssh works10:37
mmalchukthere may be problems if ~/.ssh/config exists and contain something not supported by ansible/kolla10:40
tonyhttps://paste.openstack.org/show/bqdOuNWY78wAL8xm9Xf8/10:41
tonyThat is the test I have done 10:41
tonyI am able to sssh everywhere 10:41
mmalchukyou should check from the ansible host10:42
mmalchukssh to remote 10:42
mmalchukto compute310:42
tony(kolla-openstack) stack@deployment:~$ ssh compute3 Welcome to Ubuntu 20.04.6 LTS (GNU/Linux 5.4.0-159-generic x86_64)   * Documentation:  https://help.ubuntu.com  * Management:     https://landscape.canonical.com  * Support:        https://ubuntu.com/advantage    System information as of Wed 30 Aug 2023 12:43:47 PM CAT    System load:  0.21               Swap usage:  0%       Users logged in: 0   Usage of /:   4.0% of 438.54GB   Temperature: 57.0 C 10:44
tonyIt works Ok 10:44
mmalchukok. how did you got an error ? may be a new run after ssh check would pass?10:45
mmalchukmaybe issue in key fingerprint you are not accepted before?10:46
mmalchukor whatever?10:46
mmalchukas you said you add two compute nodes10:47
mmalchukdid you configure them the same way as previous nodes?10:48
tonyI am regenerating the keys and check again 10:52
opendevreviewMerged openstack/kolla-ansible master: Add ironic-prometheus-exporter  https://review.opendev.org/c/openstack/kolla-ansible/+/87438010:56
tonyThey have also failed 10:57
mmalchukcheck the logs before, maybe you see next errors which caused by previous one11:07
mmalchukanyway this is not an ansible playbook error but something with ssh as en error shows11:07
opendevreviewMartin Hiner proposed openstack/kolla-ansible master: Add support of podman deployment  https://review.opendev.org/c/openstack/kolla-ansible/+/79922911:14
SvenKieskeregex fans invited to take a look at: https://review.opendev.org/c/openstack/kolla-ansible/+/879426 ;)11:26
kevkoHi, folks , i see this error message on grafana dashboard (kolla-ansible deployed with custom home dashboard) https://github.com/grafana/grafana/issues/54574 ...11:36
kevkowe should fix 11:36
kevkobut firstly ...do anyone know how to fix ? :) 11:37
mnasiadkaSvenKieske: I always loved those long regex strings in fluentd configs12:07
SvenKieskemnasiadka: well, they're still somewhat readable. I have seen worse :D12:23
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: ansible: bump min version to 2.14 and max to 2.15  https://review.opendev.org/c/openstack/kolla-ansible/+/88722712:24
mnasiadkatime to get back to merging this ;)12:24
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: Change regex for the multiline parser at the fluentd rabbitmq input  https://review.opendev.org/c/openstack/kolla-ansible/+/87942612:25
opendevreviewMatt Crees proposed openstack/kayobe master: CI: Migrate RabbitMQ queues during upgrade tasks  https://review.opendev.org/c/openstack/kayobe/+/89313712:25
SvenKieskeI still think I spotted a bug there, but I might be wrong, it's hard to count this stuff manually :D12:26
SvenKieskekevko: what do you mean? this is an upstream issue no? you can deploy custom dashboards like described here[0] and they should work, no?12:27
SvenKieskekevko: [0]: https://grafana.com/docs/grafana/latest/administration/provisioning/#provision-folders-structure-from-filesystem-to-grafana12:28
SvenKieskeI vaguely remember having done something like this in the past and it worked, but that might have been 2-3 years ago, so is there a bug in k-a now?12:28
sahido?12:32
SvenKieskeand a reminder to maybe look at the designate improvements, imho they are now in a rather good shape: https://review.opendev.org/c/openstack/kolla-ansible/+/87827012:33
sahido/12:33
sahidI have question regarding kolla and requirements/upper-constraints.txt12:33
sahidhow I can change from where this file is downloaded?12:33
sahidhum probably from here...12:35
SvenKieskeo/12:35
SvenKieskeI guess the answer may depend on what you really want to achieve with that :/ :D12:35
sahidwe would like to run without the need of internet12:36
sahidunfortunatly we are still on ussuri and that block kolla_toolbox_upper_constaints does not exist yet12:37
SvenKieskemhm, you could backport it, but I'm not sure how much work that would be12:42
SvenKieskeanyone know's if Jason from the Let's Encrypt Change is still active in openstack? I see no activity since last september in openstack: https://review.opendev.org/c/openstack/kolla-ansible/+/741340 https://www.stackalytics.io/?user_id=jasonandersonatuchicago&project_type=openstack&release=all&metric=marks12:43
SvenKieskesahid: if you like offline install you might also be interested in this change: https://review.opendev.org/c/openstack/kolla/+/88436812:45
opendevreviewBartosz Bezak proposed openstack/kolla-ansible stable/yoga: Added precheck for OpenSearch migration  https://review.opendev.org/c/openstack/kolla-ansible/+/89317212:46
sahidack thank you SvenKieske 12:47
kevkoSvenKieske: no it's a bug of kolla-ansible ...or let's say there has to be a way how to do it correctly if user is using custome dashboard12:48
SvenKieskewell the dashboard needs to be in place via git/docker/whatever before grafana can use it. IIRC I just altered the pipeline which produced our docker containers to include said dashboards, or did the git checkout in the grafana container, I don't remember it 100%. I still see no bug?12:49
mnasiadkamgoddard mnasiadka hrw  bbezak frickler kevko SvenKieske mmalchuk gkoper jangutter - meeting in 1012:50
kevkoSvenKieske: kolla-ansible is copying grafana_custom_home_dashboard to /etc/kolla/grafana/... on target node and config.json is rendered so it is injected into /usr/share/grafana/public/dashboards/home.json in container ... so before grafana there is already injected custom home ..stil error12:52
mnasiadkaraise a bug at least? :)12:52
SvenKieskeah, so you are saying it does not work? :D I was under the impression the linked grafana bug explains that it should work, but it's rather long and I just skimmed it12:53
kevkoSvenKieske: what i want to say is that if kolla-ansible supports custom home dashboard out-of-the box ..user shouldn't do some additional steps to fix it 12:53
kevkomnasiadka: I will, and i will also fix it :D 12:53
opendevreviewBartosz Bezak proposed openstack/kolla-ansible stable/yoga: Added precheck for OpenSearch migration  https://review.opendev.org/c/openstack/kolla-ansible/+/89317212:53
SvenKieskethinking about it, isn't /usr/share/ the wrong directory for that? but it's really been some time since I looked at it, I need to check12:54
opendevreviewMartin Hiner proposed openstack/ansible-collection-kolla master: Add Podman support  https://review.opendev.org/c/openstack/ansible-collection-kolla/+/85224012:54
SvenKieskekevko: okay, understood, I tend to agree :)12:54
kevkoSvenKieske: https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana/#default_home_dashboard_path12:54
kevkoSvenKieske: first find some information and then argument :D 12:55
SvenKieskeI just remember adding dashboards; I guess it wasn't even "home" dashboards, which seem kind of special cased12:55
SvenKieskea better path would still be /etc/grafana/ imho12:55
kevkoSvenKieske: dashboards folder working as a charm in k-a 12:56
kevkoSvenKieske: if you will read bugreport i've send you will find that home dashboard is special case ..12:56
kevkoSvenKieske: so you need to import again (the same home dashboard) via dashboard ... and error will dissapear 12:57
SvenKieskeokay, one thing that's standing out to me in the code is, that the home dashboard is owned by user root instead of grafana. but I don't know if that's on purpose, maybe grafana drops privs later?12:57
kevkoSvenKieske: even if file is present in container 12:57
SvenKieskethat's weird12:57
kevkoSvenKieske: no, it's about the database ...12:57
kevkoSvenKieske: try yourself :D 12:57
kevkoSvenKieske: if i imported same dashboard ..it disappeared 12:58
kevkoSvenKieske: one more thing i need to check is just call grafana api to reload dashboards ...i think this will work ... if it will work ..i will change the restart logic to restart -> wait -> reload dashboards 12:58
kevkothat's it 12:59
mnasiadka#startmeeting kolla13:00
opendevmeetMeeting started Wed Aug 30 13:00:04 2023 UTC and is due to finish in 60 minutes.  The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot.13:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.13:00
opendevmeetThe meeting name has been set to 'kolla'13:00
mnasiadka#topic rollcall13:00
mnasiadkao/13:00
mmalchuko/13:00
opendevreviewBartosz Bezak proposed openstack/kolla-ansible stable/yoga: Remove incorrect variable from OpenSearch role  https://review.opendev.org/c/openstack/kolla-ansible/+/89317513:00
mhinero/13:01
SvenKieskeo/13:01
bbezako/13:01
SvenKieskeit might be the case that frickler is busy with some router hardware, just pinged him privately13:03
jsuazowso/13:03
mnasiadka#topic agenda13:03
mnasiadka* CI status13:03
mnasiadka* Release tasks13:03
mnasiadka* Regular stable releases (first meeting in a month)13:03
mnasiadka* Current cycle planning13:03
mnasiadka* Additional agenda (from whiteboard)13:03
mnasiadka* Open discussion13:03
mnasiadka#topic CI status13:03
mnasiadkaoverall green I think, especially master13:03
mnasiadkaif there's anything broken - please update the whiteboard13:03
mnasiadka#topic Release tasks13:04
mnasiadkait's cycle highlights week13:04
mnasiadkaI'll try to compose something and will add let's encrypt and podman, because we should get it in13:04
SvenKieskethe LE change author seems pretty unresponsive since last september, see my message above. maybe someone could adopt the change? I guess it's only minor stuff needed13:05
kevkoi will work on comments ..13:05
mnasiadkaSvenKieske: kevko is now updating it13:05
SvenKieskenice :)13:05
mnasiadkaok, it's R-5 and we have one task13:05
mnasiadka[all] Check for new versions of infrastructure components13:05
mnasiadka#link https://docs.openstack.org/kolla/latest/contributor/release-management.html#r-5-cycle-highlights-deadline13:05
mnasiadkaanybody happy to pick this up?13:06
kevkoSvenKieske: did you diff between last patchsetset from jason and my last patchset ? :D 13:06
SvenKieskeok, time for dumb questions: what needs to be done? :D13:06
kevkoSvenKieske: there is massive change which is tested 13:06
kevkobefore it was not working properly 13:06
SvenKieskeah the link explains it13:06
mnasiadkaSvenKieske: there's a list of components listed, basically skim kolla docker images for versions and check if there are newer13:07
mnasiadkaI mean scan13:07
SvenKieskeI feel like that list is incomplete, no? if I should just check upstream I can. if this needs to be checked with supported distros etc. it's much more work.13:08
SvenKieskeis this just about upstream checking (reporting to..where?) I'm okay doing that.13:08
SvenKieskehelp appreciated, as I notice this also includes all prometheus exporters :D13:09
mnasiadkayeah, we should update that list13:09
mnasiadkaI think let's check only for versions that we control - as in prometheus exporters and similar things13:09
SvenKieskeso everything where we pull directly from upstream (repos)? still not clear to me, sorry.13:10
mnasiadkayes13:10
SvenKieskeokay, understood, seems reasonable. :)13:10
mnasiadkaso in some cases we take a repo with some major version - like in rmq/mariadb case13:10
mnasiadkaI think we bumped RMQ and MariaDB already, so it's off the list13:11
mnasiadkaceph is rather dependent on what we get from debian/ubuntu/centos repos13:11
mnasiadkaok, if you need any help - shout out :)13:11
SvenKieskecool13:11
mnasiadka#topic Current cycle planning13:12
mnasiadkaSo, Let's Encrypt - kevko will update based on comments13:12
mnasiadkaPodman - mhiner need any help, or we just need to start reviewing?13:12
mhinerYes, I need help. Can you look into whiteboard for tldr?13:13
mhinerWhat I don't understand is why this issue only happens on aarch64 debian test case13:13
mnasiadkaK-A Podman patch https://review.opendev.org/c/openstack/kolla-ansible/+/799229 failed test: https://zuul.opendev.org/t/openstack/build/781e142dcb194767a130ed3670226aa5/logs13:14
mnasiadkaSeems like the ansible insde of kolla_toolbox is unable to write in it's log file just for this single test case. Why could that be?13:14
mnasiadkakolla-ansible/ansible/roles/common/handlers/main.yml:2 fails with: [WARNING]: log file at /var/log/kolla/ansible.log is not writeable and we cannot create it, aborting13:14
mnasiadkaaarch64 only? huh13:14
SvenKieskeare we sure that the kolla_toolbox container has cap_audit? because that's a common way to not be able to write. I know the patchset enables this cap, but maybe not globally enough? just a random thought.13:15
opendevreviewBartosz Bezak proposed openstack/kolla-ansible stable/yoga: Added precheck for OpenSearch migration  https://review.opendev.org/c/openstack/kolla-ansible/+/89317213:15
mnasiadkamhiner: https://63942ee63c42f3e33db1-2a052569d3facbabacb331e67cb0b17e.ssl.cf2.rackcdn.com/799229/84/check-arm64/kolla-ansible-debian-aarch64-podman/781e142/primary/logs/container_logs/kolla_toolbox.txt - that looks a bit weird?13:15
kevkozun also failing for debian 13:15
SvenKieskemhiner: can you maybe add a simple "echo 'test' >> /var/log/kolla/ansible.log" somewhere to rule out bugs in ansible/python?13:16
SvenKieskeand maybe also an echo "test" > /tmp/testfile.txt or something to rule out permission problems with /var/log/13:17
mhinerto be clear, inside of the kolla_toolbox right?13:17
SvenKieskeunless someone has a better idea to narrow it down. I didn't take a closer look.13:17
mhinerand iirc, kolla_toolbox is in privileged mode so it should not need any additional caps13:18
SvenKieskemhiner: sure, inside the container, with the same user13:18
SvenKieskemhiner: right, if it's privileged it should suffice. but weird that it's only failing on aarch64 like mnasiadka mentioned.13:18
mnasiadkawell, basically as you can see we have no clue, but keep debugging :)13:19
mnasiadka#topic Additional agenda (from whiteboard)13:20
mnasiadkaOne was the podman one we discussed13:20
SvenKieskewe could do one recheck to rule out spurious sunflare errors in the DC :)13:20
mmalchukhave opend discussion related to podman13:20
mnasiadkaok, frickler - octavia jobboard - I think it's getting reviews13:21
mnasiadkaSvenKieske: asking for review/feedback on https://review.opendev.org/c/openstack/kolla-ansible/+/87476913:21
mnasiadkaI think we did speak about it last time, or outside of a meeting13:22
mmalchukcommented octavia jobboard13:22
fricklerack, thx mmalchuk 13:22
SvenKieskeI13:22
mnasiadkaif the author would add some CI tests and move the python inline code to a normal file - would be nice13:22
SvenKieskeI'll try to poke the author of said patch13:22
SvenKieskeI forgot to remove it from the whiteboard13:22
mnasiadkaSvenKieske: the kolla patch you wanted to talk about is already merged - so I guess that concludes :)13:23
mnasiadkak.yarovoy: podman for kolla - https://review.opendev.org/c/openstack/kolla/+/825791/10 one patchset left, but it will need some efforts on rebase and adaptation of ci. Is there any planned way to approach it?13:23
SvenKieskeyeah, didn't update the whiteboard, just did it, sorry.13:23
mnasiadkaok then, would be nice to have support for podman build in Kolla13:24
kevkobtw, all container logs are broken https://63942ee63c42f3e33db1-2a052569d3facbabacb331e67cb0b17e.ssl.cf2.rackcdn.com/799229/84/check-arm64/kolla-ansible-debian-aarch64-podman/781e142/primary/logs/container_logs/index.html13:24
mnasiadkamhiner: k.yarovoy is your colleague?13:24
opendevreviewMatt Crees proposed openstack/kayobe master: CI: Migrate RabbitMQ queues during upgrade tasks  https://review.opendev.org/c/openstack/kayobe/+/89313713:24
mhinermnasiadka: yes13:24
mnasiadkamhiner: if he could rebase the patch (now it's in merge conflict) - I can help with CI13:24
opendevreviewMatt Crees proposed openstack/kayobe master: CI: Migrate RabbitMQ queues during upgrade tasks  https://review.opendev.org/c/openstack/kayobe/+/89313713:25
mnasiadkamhiner: he's not here, so maybe you could convey the message :)13:25
mhineralso, he did not write it bet he wanted to ask if someone could pick it up bcs he won't be able to attend it for at least two next weeks13:25
mnasiadkaah ok13:25
mnasiadkaI'll try rebasing on my own then and adding CI13:25
mhinerthank you13:25
SvenKieskekevko: can you open a bug? I'm not sure what changed recently in the logging infrastructure13:26
SvenKieskesample error msg: Error: invalid argument "all" for "--tail" flag: strconv.ParseInt: parsing "all": invalid syntax13:26
mnasiadkaSvenKieske: it's the podman patch - probably logging is broken only there13:26
mnasiadkanext one:13:26
mnasiadkajsuazows - Easy bug fixes proposed:13:26
mnasiadkaOVS_DPDK task name fix13:26
mnasiadkahttps://review.opendev.org/c/openstack/kolla-ansible/+/89261313:26
mnasiadkaHorizon SSO fix13:26
mnasiadkahttps://review.opendev.org/c/openstack/kolla-ansible/+/89192313:26
SvenKieskethe horizon patch just needs people with these weird +2 powers ;)13:27
mnasiadkayeah, let's hope it doesn't break anything :)13:27
mmalchukmnasiadka another one trivial: https://review.opendev.org/c/openstack/kolla-ansible/+/89283713:27
SvenKieskeI'll look at dpdk even if I have no real interest in it.13:27
mnasiadkabut I guess I did override that somewhere locally for SSO to work13:27
mnasiadkadpdk is trivial fix13:28
SvenKieskethe SSO stuff was rather good tested, afaik. don't know if the github issue is linked somewhere13:28
mnasiadkabbezak: willing to have a look in the SSO patch? ^^13:28
kevkomnasiadka: dpdk approved13:30
mmalchukand server-status and kayobe patches from the last meeting question?)13:30
SvenKieskenvm, I thought about another SSO bug, rather that one was about OIDC..13:31
mnasiadkaok, those are done13:31
mnasiadkammalchuk: can you post a link so other cores can have a look?13:31
mmalchuk#link https://review.opendev.org/c/openstack/kayobe/+/86139713:32
mmalchuk#link https://review.opendev.org/c/openstack/kayobe/+/87955413:32
opendevreviewJan Gutter proposed openstack/kolla-ansible master: etcd: update to v3.4  https://review.opendev.org/c/openstack/kolla-ansible/+/89046513:32
mmalchuk#link https://review.opendev.org/c/openstack/kolla/+/89109813:32
mnasiadkaok then13:34
kevkohttps://review.opendev.org/q/owner:lukas.mrtvy%2540gmail.com << another sso fixes abandoned ... guesswhat8 abandoned13:34
mnasiadka#topic Open discussion13:34
mmalchuk1. "bad practice"13:34
mmalchukhttps://review.opendev.org/c/openstack/ansible-collection-kolla/+/89299013:34
mmalchukunset or comment?13:34
mnasiadkaI have no preference, but it seems like a cosmetic change with no gain13:37
mmalchukimho default+omit is a good practice13:37
mnasiadkaas long as we don't have coding guidelines with best practices documented and align all code to this - I don't think it makes any sense13:37
fricklergain of good style, I'm +2 and we can add that to coding guidelines?13:38
SvenKieske{{ virtualenv is none | ternary(omit, virtualenv) }}" is way harder to read than: {{ virtualenv | default(omit) }}13:38
mnasiadkado we have any coding guidelines? :)13:38
fricklerdidn't we link to that last week?13:39
mnasiadkawell, ternary is not there today, it's only afaik in podman patch13:39
SvenKieskewe could invent some :) I'm all for it, because some reviews boil down to taste. I think a guideline can help cut discussions short13:39
SvenKieskeit's the next best thing to: always do what the linter tells you13:39
mmalchukanyone have a link to coding guidelines?13:39
mnasiadkafirst of all, that's not the only occurence of using "is not none" for an empty defined var in ansible-collection-kolla13:40
mnasiadkaso either we change all of them or none13:40
fricklerhttps://docs.openstack.org/kolla/latest/contributor/kolla-design-philosophy.html , maybe need to extend the domain from design to coding?13:40
SvenKieskelike the python "black" approach, or gofmt, or rustfmt13:40
mmalchuk+1 for all of them)13:40
mnasiadkasecond of all, maybe newer ansible-lint at least warns on something like that, then it would be easier to enforce13:40
mnasiadkaor we need to write a custom rule13:40
SvenKieskeso let me rephrase: I don't like coding guidelines, I like syntax enforced by a linter (basically coding guidelines automated as code). If I can't have that; I like guidelines.13:41
mmalchukansible-lint quiet about this13:41
mnasiadkawe use some old version13:41
mnasiadkamaybe it's time to bump up13:41
mmalchukmay be never version?13:41
mnasiadkaand then add some custom rules if needed13:41
SvenKieskeansible-lint is really a pain because they, at least sometimes, change quite a lot of stuff which results in new errors and warnings13:42
fricklermaybe that cannot be caught by a linter, because it needs too much semantic understanding13:42
SvenKieskemnasiadka: put ansible-lint on the "to check for updates" list? ;)13:42
mnasiadkaSvenKieske: might be, but that will be painful13:42
mmalchukso lets merge https://review.opendev.org/c/openstack/ansible-collection-kolla/+/892990 and fix all other places later with lint13:43
mnasiadkaok, anyway - I think linter that enforces it is the way13:43
mnasiadkammalchuk: I disagree13:43
mmalchukwhy?13:43
mmalchukstarting from coding guidelines and lint ?13:44
mnasiadkaI agree with SvenKieske that writing coding guidelines is a waste of resources, it's better to do proper linting.13:45
mnasiadkaI'm fine with fixing bad practices, but we need to check for them and fix them everywhere, not in just one role.13:45
mmalchukupgrade lint and make job non voting?13:45
mnasiadkaupgrade lint and stack lint update on top of patches that make us compliant13:45
SvenKieskeI don't know if a tool can enforce good style on playbooks, the ansible-lint stuff is okayish, but had some drawbacks in the past. as always: a linter or coding guidelines will incur additional work13:46
SvenKieskebut it hopefully will make stuff more consistent and maybe also catch some errors13:46
mmalchukdo we really have linter like black ?13:46
mmalchukwhich changes the code13:46
SvenKieskeansible-lint --write . 13:47
SvenKieskeI don't know how reliable that is, but it exists :)13:47
mmalchukwow... thanks. didn't know13:48
mmalchukwill check all the code)))13:48
mnasiadkaok then13:50
SvenKieskeas long as we realize that everything in live has also drawbacks it might be a worthwhile extension :) 13:50
mnasiadkaanything else?13:51
jangutterfans of ARA logging | https://review.opendev.org/c/openstack/kolla-ansible/+/891097  :-p13:51
mmalchuklack of kayobe cores13:51
mnasiadkajangutter: I'm still not convinced around the additional time of jobs13:52
SvenKieskeyou can scratch kayobe from that sentence ;)13:52
mmalchuk:)13:52
mmalchukanyway13:52
SvenKieskeregarding ARA: It might be nice to be able to enable it on a case-by-case basis?13:52
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Add support of podman deployment  https://review.opendev.org/c/openstack/kolla-ansible/+/79922913:53
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Fix podman logs  https://review.opendev.org/c/openstack/kolla-ansible/+/89318713:53
janguttermnasiadka : that is indeed a fair reason not to do it. I can easily change it so that it can be triggered by something in the commit message.13:53
tonyHello Team, .. I am facing the issue regarding installing new compute nodes on my wallaby openstack cluster using kolla-ansible, I am getting the error below13:54
mnasiadkawe might go that path, ara might be sometimes useful13:54
mnasiadkatony: wait a moment, there's a meeting in progress13:54
mnasiadkajangutter: can you rework that please?13:54
tonyOooh My Bad13:54
mmalchukbtw hell of ansible-lint in bifrost: https://review.opendev.org/q/project:openstack/bifrost+topic:ansible-lint13:54
janguttermnasiadka: ack, won't take too long - we happy with the interface where "#log_ara" or something in the commit message triggers it?13:55
mnasiadkajangutter: I think so13:55
mnasiadkajangutter: we could add docs around how to use it somewhere in Kolla-Ansible contributor guide?13:55
janguttermnasiadka: good idea, I'll add it there!13:56
mnasiadkathanks!13:56
mnasiadkaok, I guess it's enough for today :)13:56
mnasiadkaThanks all for coming!13:56
mnasiadka#endmeeting13:56
opendevmeetMeeting ended Wed Aug 30 13:56:27 2023 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)13:56
opendevmeetMinutes:        https://meetings.opendev.org/meetings/kolla/2023/kolla.2023-08-30-13.00.html13:56
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/kolla/2023/kolla.2023-08-30-13.00.txt13:56
opendevmeetLog:            https://meetings.opendev.org/meetings/kolla/2023/kolla.2023-08-30-13.00.log.html13:56
SvenKieskethanks for all your efforts o/13:58
tony@mnasiadka, Would now be a good time to share my findings ?14:00
mnasiadkatony: yes, but for pasting logs/code/whatever please use paste.openstack.org14:00
tonySure 14:01
tonySo I can confirm that the ssh is not the issue as for other services / containers the ssh works minus for toolbox, fluentd and cron 14:02
tonyi also found out that on the new compute nodes, they have docker version 24 after doing the bootstrap but the older ones have 20.14:02
kevkotony: probably because state=present in ansible ? :D 14:04
tony@kevko, I didnt manipulate anything in ansible 14:05
tonyBut my question is... Could that be the reason for the failure ?14:05
mmalchuklost my internet connection14:05
mmalchukmnasiadka thanks14:06
opendevreviewBartosz Bezak proposed openstack/kolla-ansible stable/yoga: Remove incorrect variable from OpenSearch role  https://review.opendev.org/c/openstack/kolla-ansible/+/89317514:09
opendevreviewMerged openstack/kayobe master: Speed up calls to Bifrost  https://review.opendev.org/c/openstack/kayobe/+/88295114:11
kevkotony where is your log or something ?  14:13
tonyLet me share14:14
kevkotony use paste.openstack.org and share the link 14:14
opendevreviewBartosz Bezak proposed openstack/kolla-ansible stable/yoga: Added precheck for OpenSearch migration  https://review.opendev.org/c/openstack/kolla-ansible/+/89317214:15
tonyHere is the link14:15
tonyhttps://paste.openstack.org/show/bNxHwZb4yqmTFUX7ulX6/14:15
opendevreviewBartosz Bezak proposed openstack/kolla-ansible stable/yoga: Added precheck for OpenSearch migration  https://review.opendev.org/c/openstack/kolla-ansible/+/89317214:16
kevkotony: incompatibility between docker and kolla module14:16
kevkotony: you need to backport one commit to your downstream kolla-ansible repository 14:16
kevkotony: or amend your kolla-ansible code on disk 14:17
kevkolet me check 14:17
kevkotony: do you have this change applied ? https://review.opendev.org/c/openstack/kolla-ansible/+/87267114:18
tonyLet me check 14:19
tony@kevko, Am sorry but I am abit on the change and unaware on how to change it, Any guidance will be appreciated 14:22
kevkogrep KernelMemory your_path_for_kolla_ansible_code/ansible/library/kolla_docker.py14:23
kevkotony: ^^14:23
tonyon it 14:24
kevkoif grep -rq self._dimensions_kernel_memory_removed    your_path_for_kolla_ansible_code/; then echo "commit included"; else echo "commit not included";fi14:25
tonyresults of the  grep KernelMemory your_path_for_kolla_ansible_code/ansible/library/kolla_docker.py14:29
tonyhttps://paste.openstack.org/show/bDVCZYLpkXbzEEnfdHTz/14:29
kevkotony:  well, you have old version of kolla-ansible code14:30
kevkotony: just update your kolla-ansible code and it will work 14:30
tonyWould this command pip update ./kolla-ansible be enough ?14:33
opendevreviewPierre Riteau proposed openstack/kayobe stable/2023.1: Speed up calls to Bifrost  https://review.opendev.org/c/openstack/kayobe/+/89320114:33
opendevreviewPierre Riteau proposed openstack/kayobe stable/zed: Speed up calls to Bifrost  https://review.opendev.org/c/openstack/kayobe/+/89320214:33
opendevreviewPierre Riteau proposed openstack/kayobe stable/yoga: Speed up calls to Bifrost  https://review.opendev.org/c/openstack/kayobe/+/89320314:33
opendevreviewPierre Riteau proposed openstack/kayobe stable/xena: Speed up calls to Bifrost  https://review.opendev.org/c/openstack/kayobe/+/89320414:33
opendevreviewPierre Riteau proposed openstack/kayobe stable/wallaby: Speed up calls to Bifrost  https://review.opendev.org/c/openstack/kayobe/+/89320514:34
opendevreviewMerged openstack/kolla-ansible master: Fixes task name in notify module to the actual task name  https://review.opendev.org/c/openstack/kolla-ansible/+/89261314:49
opendevreviewMerged openstack/kolla-ansible master: Drop useless DUMMY_ENVIRONMENT environment for cron container  https://review.opendev.org/c/openstack/kolla-ansible/+/89283714:49
opendevreviewMerged openstack/kolla-ansible master: Fixes WEBSSO_KEYSTONE_URL Value  https://review.opendev.org/c/openstack/kolla-ansible/+/89192314:50
opendevreviewVerification of a change to openstack/kolla-ansible stable/yoga failed: opensearch: alter path after using rpm/deb packaging  https://review.opendev.org/c/openstack/kolla-ansible/+/88437614:52
opendevreviewMaksim Malchuk proposed openstack/kolla-ansible stable/2023.1: Fixes task name in notify module to the actual task name  https://review.opendev.org/c/openstack/kolla-ansible/+/89320614:59
opendevreviewMaksim Malchuk proposed openstack/kolla-ansible stable/zed: Fixes task name in notify module to the actual task name  https://review.opendev.org/c/openstack/kolla-ansible/+/89320714:59
opendevreviewMaksim Malchuk proposed openstack/kolla-ansible stable/2023.1: Drop useless DUMMY_ENVIRONMENT environment for cron container  https://review.opendev.org/c/openstack/kolla-ansible/+/89320815:02
opendevreviewMaksim Malchuk proposed openstack/kolla-ansible stable/zed: Drop useless DUMMY_ENVIRONMENT environment for cron container  https://review.opendev.org/c/openstack/kolla-ansible/+/89320915:02
opendevreviewMaksim Malchuk proposed openstack/kolla-ansible stable/yoga: Drop useless DUMMY_ENVIRONMENT environment for cron container  https://review.opendev.org/c/openstack/kolla-ansible/+/89321015:02
kevkotony: no 15:05
kevkotony: for wallaby install wallaby version from git 15:05
kevkofrom stable branch 15:05
alistarlehello guy's, we are lacking a second +2 on our fixup patch for deploying ironic in kolla, if you think you can merge it :) https://review.opendev.org/c/openstack/kolla-ansible/+/88674715:06
opendevreviewMaksim Malchuk proposed openstack/kolla-ansible stable/2023.1: Fixes WEBSSO_KEYSTONE_URL Value  https://review.opendev.org/c/openstack/kolla-ansible/+/89321215:06
opendevreviewMaksim Malchuk proposed openstack/kolla-ansible stable/zed: Fixes WEBSSO_KEYSTONE_URL Value  https://review.opendev.org/c/openstack/kolla-ansible/+/89321315:07
opendevreviewMaksim Malchuk proposed openstack/kolla-ansible stable/2023.1: Fix D001 Line too long in mariadb-guide  https://review.opendev.org/c/openstack/kolla-ansible/+/89321415:15
opendevreviewMaksim Malchuk proposed openstack/kolla-ansible stable/zed: Fix D001 Line too long in mariadb-guide  https://review.opendev.org/c/openstack/kolla-ansible/+/89321515:16
SvenKieskemnasiadka: imho this needs relnotes as suddenly ironic-tftp is not working on all IPs anymore? https://review.opendev.org/c/openstack/kolla-ansible/+/88674715:16
opendevreviewMaksim Malchuk proposed openstack/kolla-ansible stable/yoga: Fix D001 Line too long in mariadb-guide  https://review.opendev.org/c/openstack/kolla-ansible/+/89321615:16
opendevreviewMaksim Malchuk proposed openstack/kolla-ansible stable/xena: Fix D001 Line too long in mariadb-guide  https://review.opendev.org/c/openstack/kolla-ansible/+/89321715:16
SvenKieskeI replied in the (merged) patch discussion, sometimes we are too fast imho, when it comes to merging: https://review.opendev.org/c/openstack/kolla-ansible/+/88674715:39
SvenKiesketo be fair, the patchset is not that new, but I didn't notice it until now. my bad I guess.15:40
kevkodrop release notes :D 15:45
tonyHey Kevko15:54
tonyThe upgrade helped15:54
tonyThanks a lot 15:54
SvenKieskekevko: I was bitten enough times by missing relnotes for crucial stuff just changing without any notice in openstack. I think we should and can do better. or just give up and publish a git log for relnotes16:05
SvenKieskebut we had that discussion already some weeks ago.16:06
SvenKieskehttps://keepachangelog.com16:06
SvenKieskethis is basically https://semver.org/ for changelogs/release notes.16:07
opendevreviewMerged openstack/kolla master: Add server-status handler to Rocky/Centos Apache conf  https://review.opendev.org/c/openstack/kolla/+/89109816:08
opendevreviewMaksim Malchuk proposed openstack/kolla stable/2023.1: Add server-status handler to Rocky/Centos Apache conf  https://review.opendev.org/c/openstack/kolla/+/89322016:09
opendevreviewMaksim Malchuk proposed openstack/kolla stable/zed: Add server-status handler to Rocky/Centos Apache conf  https://review.opendev.org/c/openstack/kolla/+/89324116:09
opendevreviewMaksim Malchuk proposed openstack/kolla stable/yoga: Add server-status handler to Rocky/Centos Apache conf  https://review.opendev.org/c/openstack/kolla/+/89324216:09
opendevreviewMaksim Malchuk proposed openstack/kolla stable/xena: Add server-status handler to Rocky/Centos Apache conf  https://review.opendev.org/c/openstack/kolla/+/89324316:09
opendevreviewFelipe Figueroa Vergara proposed openstack/kolla master: Adds TAAS Neutron plugin to support OVS port mirrors  https://review.opendev.org/c/openstack/kolla/+/88515116:14
*** ralonsoh is now known as ralonsoh_ooo16:20
opendevreviewVerification of a change to openstack/kolla-ansible stable/yoga failed: opensearch: alter path after using rpm/deb packaging  https://review.opendev.org/c/openstack/kolla-ansible/+/88437616:24
opendevreviewMerged openstack/kolla-ansible master: Use better default bind address for ironic-tftp  https://review.opendev.org/c/openstack/kolla-ansible/+/88674717:03
opendevreviewHao Wang proposed openstack/kolla-ansible master: Support deploy Zaqar with kolla-ansible  https://review.opendev.org/c/openstack/kolla-ansible/+/89261517:20

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!