| opendevreview | Jorhson Deng proposed openstack/kolla master: Unify curl use https://review.opendev.org/c/openstack/kolla/+/902514 | 01:09 |
|---|---|---|
| opendevreview | Jorhson Deng proposed openstack/kolla master: Unify curl use https://review.opendev.org/c/openstack/kolla/+/902514 | 01:29 |
| opendevreview | wu.chunyang proposed openstack/kolla-ansible master: CI: Trove: Create a database instance https://review.opendev.org/c/openstack/kolla-ansible/+/863521 | 03:25 |
| opendevreview | wu.chunyang proposed openstack/kolla-ansible master: CI: Trove: Create a database instance https://review.opendev.org/c/openstack/kolla-ansible/+/863521 | 06:21 |
| opendevreview | Jorhson Deng proposed openstack/kolla master: Unify curl use https://review.opendev.org/c/openstack/kolla/+/902514 | 06:37 |
| SvenKieske | o/ | 08:04 |
| SvenKieske | dcapone2004: glad you could work it out, what was the issue, if I may ask? :) | 08:04 |
| opendevreview | Pierre Riteau proposed openstack/kayobe master: Pin ansible-core at 2.15.6 https://review.opendev.org/c/openstack/kayobe/+/902614 | 08:15 |
| mnasiadka | frickler: do we want to merge gnocchi patch in Kolla before final release? | 08:28 |
| mnasiadka | https://review.opendev.org/c/openstack/kolla/+/901574?tab=change-view-tab-header-zuul-results-summary | 08:29 |
| frickler | mnasiadka: we can merge it, the question is whether we want to do a backport and an rc2? though I think we need the latter anyway due to bifrost/kayobe? | 08:32 |
| mnasiadka | frickler: yes, we need rc2 for kolla | 08:37 |
| frickler | ok, so let's add gnocchi to that | 08:40 |
| opendevreview | Merged openstack/kolla master: Revert "Disable build of Gnocchi containers" https://review.opendev.org/c/openstack/kolla/+/901574 | 10:30 |
| frickler | mnasiadka: mgoddard: what about https://review.opendev.org/c/openstack/kolla-ansible/+/805598 ? at least let's merge it into master now and we can look at backporting later? | 10:43 |
| mnasiadka | well, that has merge conflict - so once that's solved - sure | 10:44 |
| frickler | that's what I was pinging mgoddard for ;) though it seems kevko did the last rebase? sigh, let me check it myself then | 10:47 |
| opendevreview | Dr. Jens Harbott proposed openstack/kolla-ansible master: Support Ansible max_fail_percentage https://review.opendev.org/c/openstack/kolla-ansible/+/805598 | 10:52 |
| frickler | ah, that was an easy one for once | 10:52 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/2023.2: Revert "Disable build of Gnocchi containers" https://review.opendev.org/c/openstack/kolla/+/902539 | 10:59 |
| mnasiadka | ok then, this merges and then rc2 | 11:00 |
| andreykurilin | Hi folks! Can someone help me understand how properly override horizon policies with kolla-ansible of 2023.1+ ? I think I’m facing some bug (regression of ansible) which is also reproducible in CI | 11:03 |
| opendevreview | Pierre Riteau proposed openstack/kayobe master: Fix Ansible failure with new ansible-core https://review.opendev.org/c/openstack/kayobe/+/902634 | 11:04 |
| frickler | andreykurilin: where do you see the failure in the ci? | 11:18 |
| andreykurilin | frickler: I made a small commit to try reproduce the issue https://review.opendev.org/c/openstack/kolla-ansible/+/901761 | 11:26 |
| andreykurilin | frickler: I have not submitted a bug report yet (idk whether it should be submitted to kolla-ansible or Ansible itself) | 11:26 |
| andreykurilin | It looks like a regression of https://github.com/ansible/ansible/issues/39389 . I can reproduce it with min required ansible of kolla-ansible 2023.1 and latest version as well | 11:28 |
| andreykurilin | https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_2c0/901761/1/check/kolla-ansible-ubuntu/2c06b57/primary/logs/ansible/deploy here you can grep by ansible.errors.AnsibleFileNotFound | 11:30 |
| andreykurilin | currently, we are bypassing this issue in our deployment by hacking kolla-ansible to use older ansible release as a short-term solution. It is ugly way, so want to hear whether we are doing something completely wrong, or it looks like a valid bug, etc | 11:32 |
| frickler | andreykurilin: using the example from the issue I get "foo": "[] + [ 'one' ] + [ 'two' ] + [ 'three' ]" with ansible 2.15.2, so this seems to have indeed regressed again, but different than before | 12:15 |
| frickler | mnasiadka: ^^ not sure yet if this is a usage issue on our side or a real ansible bug, though? | 12:16 |
| SvenKieske | did anybody already (re)opened a bug either in k-a or upstream about this? | 12:46 |
| frickler | SvenKieske: andreykurilin: seems https://github.com/ansible/ansible/issues/80404 is the answer. so bug in k-a | 12:48 |
| SvenKieske | mhm, do we use this pattern often? oO | 12:52 |
| frickler | I'm wondering that, too. see also https://docs.ansible.com/ansible-core/2.14/porting_guides/porting_guide_core_2.13.html#playbook | 12:52 |
| andreykurilin | frickler: thank you for pointing to Ansible’s bug report and docs. I couldn't find it quickly | 12:55 |
| frickler | andreykurilin: thanks for providing the original issue, I just searched for the title of that once more ;) | 12:56 |
| frickler | andreykurilin: do you want to create the bug report for kolla-ansible since you discovered the issue? the fix should be pretty easy then, but we'll want to backport so having a bug attached will be better | 12:58 |
| andreykurilin | SvenKieske: it looks like this pattern is used only in one place - horizon role. | 12:58 |
| andreykurilin | frickler: working on it | 12:58 |
| mnasiadka | phew, situation solved itself :) | 13:00 |
| frickler | two easy issues in a row, what kind of holiday is this? ;) | 13:04 |
| andreykurilin | https://bugs.launchpad.net/kolla-ansible/+bug/2045660 I am not good at writing, but tried to put enough info | 13:08 |
| SvenKieske | ty! | 13:14 |
| mnasiadka | so where's the fix? :) | 13:18 |
| mnasiadka | hmm | 13:20 |
| mnasiadka | 2023-12-05 12:59:09.582 971 ERROR oslo_service.service amqp.exceptions.PreconditionFailed: Queue.declare: (406) PRECONDITION_FAILED - inequivalent arg 'x-queue-type' for queue 'cinder-volume' in vhost '/': received the value 'quorum' of type 'longstr' but current is none | 13:20 |
| mnasiadka | in https://zuul.opendev.org/t/openstack/build/7a5631bcfd0b4c2cb7d2e4d7f1dcff8e/logs | 13:21 |
| andreykurilin | mnasiadka: I can work on it, but as @frickler found the solution, it makes sense to give all the credits to him | 13:23 |
| mnasiadka | You can add "Co-Authored-By: " in the commit message, I'm sure frickler will be happy ;) | 13:24 |
| andreykurilin | Ok, start working on it | 13:25 |
| * frickler sure is happy, will happily approve such a patch :) | 13:25 | |
| opendevreview | Andriy Kurilin proposed openstack/kolla-ansible master: Fix broken list concatenation in horizon role https://review.opendev.org/c/openstack/kolla-ansible/+/902659 | 13:45 |
| opendevreview | Andriy Kurilin proposed openstack/kolla-ansible master: Fix broken list concatenation in horizon role https://review.opendev.org/c/openstack/kolla-ansible/+/902659 | 13:50 |
| opendevreview | Andriy Kurilin proposed openstack/kolla-ansible master: Fix broken list concatenation in horizon role https://review.opendev.org/c/openstack/kolla-ansible/+/902659 | 13:59 |
| opendevreview | Pierre Riteau proposed openstack/kayobe stable/2023.1: Pin ansible-core at 2.14.11 https://review.opendev.org/c/openstack/kayobe/+/902660 | 14:00 |
| opendevreview | Merged openstack/kolla stable/2023.2: Revert "Disable build of Gnocchi containers" https://review.opendev.org/c/openstack/kolla/+/902539 | 15:06 |
| mnasiadka | ok, gnocchi is there, time for rc2 | 15:21 |
| andreykurilin | https://review.opendev.org/c/openstack/kolla-ansible/+/902659 upgrade jobs are failing due to new steps that I introduced. I do not know how upgrade jobs are designed, so I am wondering whether moving tests into a separate follow-up change will fix an issue or not. Any suggestions? | 15:25 |
| opendevreview | Pierre Riteau proposed openstack/kayobe master: Install setuptools before creating virtualenv https://review.opendev.org/c/openstack/kayobe/+/881957 | 15:26 |
| SvenKieske | mhhm | 15:28 |
| SvenKieske | andreykurilin: are you sure the failure is related to your change? unfortunately the arm64 pipeline fails regularly for all kind of reasons. | 15:29 |
| frickler | we need to backport the fix to 2023.1 before the upgrade jobs work I think | 15:32 |
| frickler | andreykurilin: so maybe split the fix and the new tests into two patches, we can backport the fix first then | 15:33 |
| SvenKieske | ah of course | 15:34 |
| frickler | also we need to change master to upgrade from 2023.2 now? or do we wait for the official release first? | 15:35 |
| SvenKieske | mhm, isn't that specified in our release docs? I guess it isn't? | 15:36 |
| SvenKieske | would there be any negative side effect by already switching now? | 15:36 |
| SvenKieske | I can't think of any right now, but I never thought about this problem before. | 15:37 |
| mnasiadka | frickler: actually - did we update master upgrade jobs to use 2023.1? | 15:44 |
| mnasiadka | erm 2023.2 | 15:45 |
| mnasiadka | but the images in master are 2023.2 | 15:45 |
| mnasiadka | so the upgrade is not an upgrade now :D | 15:45 |
| mnasiadka | well, we upgrade from 2023.1 to 2023.2 (master) | 15:45 |
| mnasiadka | sorry, my brain probably decided to shut off | 15:45 |
| mnasiadka | let's get 2023.2 released as final, then we can do some crazy things | 15:46 |
| SvenKieske | what about the slurp upgrade stuff? there are no volunteers to implement that yet, right? (sorry for making stuff even more complicated) | 15:46 |
| mnasiadka | normally we did not switch in the past for a couple of weeks, to get some stability | 15:46 |
| mnasiadka | SvenKieske: SHPC is going to work on SLURP | 15:46 |
| SvenKieske | :) | 15:46 |
| SvenKieske | so wait for some stability it is? | 15:48 |
| mnasiadka | I think so, let's release final for bobcat | 15:48 |
| mnasiadka | and then think what next tomorrow | 15:48 |
| opendevreview | Merged openstack/kolla-ansible master: Support Ansible max_fail_percentage https://review.opendev.org/c/openstack/kolla-ansible/+/805598 | 15:55 |
| opendevreview | Andriy Kurilin proposed openstack/kolla-ansible master: Fix broken list concatenation in horizon role https://review.opendev.org/c/openstack/kolla-ansible/+/902659 | 16:01 |
| opendevreview | Andriy Kurilin proposed openstack/kolla-ansible master: [ci] check ability to override horizon policies https://review.opendev.org/c/openstack/kolla-ansible/+/901761 | 16:01 |
| frickler | have we ever seen module failures with error -13? just found out about https://review.opendev.org/c/openstack/openstack-ansible/+/851426 and similar issues in opendev | 16:14 |
| mnasiadka | not really | 16:15 |
| mnasiadka | hmm, there are those issues with host unreachable or something similar | 16:16 |
| mnasiadka | but those are extremely rare | 16:16 |
| SvenKieske | well not in the cephadm upgrade jobs? it's rather common there? and these are the only jobs that could trigger host unreachable? | 16:20 |
| SvenKieske | could be worth it to check if cephadm jobs are more stable with this imho. | 16:21 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Use ControlPersist and ControlMaster https://review.opendev.org/c/openstack/kolla-ansible/+/902699 | 16:24 |
| mnasiadka | let's see | 16:24 |
| SvenKieske | at least the ubuntu cephadm job is green, no failure so far :) | 17:55 |
| frickler | but again some rmq queue declare failure in cinder during upgrade. I get the feeling that our handling of the queue changes is not working really well https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_09c/902699/1/check/kolla-ansible-ubuntu-upgrade-cephadm/09cdc7f/primary/logs/kolla/all-ERROR.txt | 18:59 |
| mnasiadka | frickler: wonder that's only cinder | 19:15 |
| mnasiadka | frickler: I'll have a look tomorrow | 19:15 |
| mnasiadka | frickler: I have a feeling we could be better at checking MariaDB/RabbitMQ state after we upgrade it | 19:18 |
| opendevreview | Merged openstack/kayobe stable/2023.1: Pin ansible-core at 2.14.11 https://review.opendev.org/c/openstack/kayobe/+/902660 | 19:48 |
| dcapone2004 | SvenKieske, my issue was I had ran the "certificates" command at some point trying to fix the issue when I had the initial nova/neutron problem....what I didn't know was that that command overwrote existing certs at the paths specified in globals.yml | 21:26 |
| dcapone2004 | so everytime I kept copying what I thought was "our cert" to haproxy.pem, I was actually copying the generated cert that overwrote our cert (but because the filename didn't change, I didn't realize that is what KA did) | 21:27 |
| dcapone2004 | I thought KA would only generate certificates that DIDN'T EXIST already at the paths specified in the globals.yml file | 21:28 |
| dcapone2004 | as a side note, that behavior makes more it tricky to use a hybrid of self-signed and CA issued certs | 21:32 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!