opendevreview | Dr. Jens Harbott proposed openstack/kolla-ansible master: DNM: Failing a job to hold a node https://review.opendev.org/c/openstack/kolla-ansible/+/859214 | 06:24 |
---|---|---|
mnasiadka | morning | 07:26 |
frickler | do we need to do something special in infra to set up new container images? docker.io/kolla/prometheus-ovn-exporter:master-ubuntu-jammy exists and I can pull it, but mirror.iad3.inmotion.opendev.org:4447/openstack.kolla/prometheus-ovn-exporter:master-ubuntu-jammy says "not authorized" | 07:48 |
mnasiadka | well, this is just a proxy in Apache, right? | 07:48 |
frickler | hmm, is it? /me goes to check logs | 07:49 |
mnasiadka | and we are rather going to quay.io? | 07:49 |
mnasiadka | https://quay.io/repository/openstack.kolla/prometheus-ovn-exporter?tab=tags&tag=latest | 07:50 |
mnasiadka | but it's there, 4 hours ago | 07:50 |
frickler | ah, o.k., the pull for that gives "not authorized" to me, too. the same for e.g. mysqld exporter works | 07:51 |
frickler | seems I need a Redhat login in order to be able to view that page? | 07:53 |
mnasiadka | ah, that's interesting | 07:56 |
mnasiadka | for some reason the repo was marked as private | 07:57 |
frickler | right, https://quay.io/repository/openstack.kolla/prometheus-mysqld-exporter?tab=tags&tag=latest works without login | 07:57 |
mnasiadka | I changed it to public | 07:58 |
mnasiadka | but it seems we have 7 private repos | 07:58 |
mnasiadka | let me check | 07:58 |
frickler | the ovn-exporter pull is working now, thx mnasiadka | 08:00 |
mnasiadka | ok, getting permissions of all repos, might take a while (need to do it via API) | 08:01 |
mnasiadka | yoctozepto: seems new repositories created by docker push are getting marked as private - I think we need to extend the publish playbooks to ensure repos are public... | 08:03 |
priteau | Good morning. I think ansible-lint released on Friday broke kolla-ansible CI | 08:07 |
priteau | https://2ee6fa08daa1e3595342-a6ec19ba3cffa389cb7fa75d69a8d9a0.ssl.cf2.rackcdn.com/859113/2/check/openstack-tox-linters/be728b9/job-output.txt | 08:07 |
priteau | name: Jinja templates should only be at the end of 'name' (name[template]) | 08:08 |
priteau | why oh why | 08:08 |
priteau | Discovered in https://review.opendev.org/c/openstack/kolla-ansible/+/859113 which broke between PS1 and PS2 hours apart :( | 08:09 |
mnasiadka | doesn't seem like hard to fix though ;-) | 08:09 |
priteau | This is the docs about it: https://ansible-lint.readthedocs.io/rules/name/ | 08:10 |
priteau | name[template] - Jinja templates should only be at the end of ‘name’. This helps with the identification of tasks inside the source code when they fail. The use of templating inside name keys is discouraged as there are multiple cases where the rendering of the name template is not possible. | 08:10 |
priteau | mnasiadka: Are you ok with adding this rule to skip_list? | 08:10 |
priteau | I don't think we have issues with rendering of the name template | 08:11 |
mnasiadka | If it takes days to fix, then yes | 08:11 |
mnasiadka | but I assume we do loop on templating | 08:11 |
priteau | It fails on things like this: | 08:14 |
priteau | - name: Check if the folder ["{{ node_custom_config }}/ceilometer/{{ ceilometer_dynamic_pollsters_local_folder }}"] for dynamic pollsters definitions exist | 08:14 |
priteau | I really don't see how putting it at the end of the name field makes it better? | 08:15 |
priteau | I see, because it would truncate anything after | 08:16 |
priteau | Not a big issue really | 08:16 |
mnasiadka | well, I'm not a fan of adding things to skip_list forever, we already do have quite a collection of skip_list - if it's easy to fix it, then we should just fix it | 08:18 |
priteau | We must not bow to the ansible-lint dictatorship! | 08:19 |
priteau | Or we will do a follow up | 08:20 |
opendevreview | Pierre Riteau proposed openstack/kolla-ansible master: CI: allow Jinja templating anywhere inside task and play names https://review.opendev.org/c/openstack/kolla-ansible/+/859220 | 08:20 |
mnasiadka | frickler: seems it was the only private repository | 08:25 |
priteau | CI checks passed ^^^ | 08:30 |
opendevreview | Merged openstack/kolla-ansible master: CI: allow Jinja templating anywhere inside task and play names https://review.opendev.org/c/openstack/kolla-ansible/+/859220 | 08:48 |
opendevreview | Pierre Riteau proposed openstack/kolla-ansible master: Fix Ironic API healthcheck with backend TLS https://review.opendev.org/c/openstack/kolla-ansible/+/859222 | 08:52 |
opendevreview | Pierre Riteau proposed openstack/kolla-ansible master: Fix interface option for ironic-neutron-agent https://review.opendev.org/c/openstack/kolla-ansible/+/859113 | 08:52 |
opendevreview | Ivan Halomi proposed openstack/kolla-ansible master: Add support of podman deployment https://review.opendev.org/c/openstack/kolla-ansible/+/799229 | 09:04 |
opendevreview | Michal Nasiadka proposed openstack/kolla master: Add Opensearch image(s) https://review.opendev.org/c/openstack/kolla/+/830373 | 09:15 |
opendevreview | Verification of a change to openstack/kayobe master failed: Update to latest CentOS Stream 8 cloud image https://review.opendev.org/c/openstack/kayobe/+/857869 | 09:22 |
opendevreview | Michal Nasiadka proposed openstack/kolla master: Add Opensearch image(s) https://review.opendev.org/c/openstack/kolla/+/830373 | 09:32 |
opendevreview | Marcin Juszkiewicz proposed openstack/kolla-ansible master: ansible-lint: cleanup our skip list https://review.opendev.org/c/openstack/kolla-ansible/+/859252 | 09:44 |
hrw | as we do ansible-lint cleanups... | 09:44 |
mnasiadka | hrw: we don't have baremetal role anymore, so maybe we don't need all of them ;) | 09:46 |
hrw | you are too optimistic ;D | 09:46 |
hrw | Finished with 1676 failure(s), 9 warning(s) on 1774 files. (without .ansible-lint) | 09:47 |
hrw | ok. who will review centos 9 patch? https://review.opendev.org/c/openstack/kolla/+/836664 | 09:49 |
hrw | as it feels abandoned | 09:50 |
admin1 | i added a component ( octavia ) in an existing cluster and the playbooks completed . but horizon shows internal server error .. how do I troubleshoot it ? | 09:59 |
opendevreview | Pierre Riteau proposed openstack/kayobe master: [release] Set previous_release to yoga https://review.opendev.org/c/openstack/kayobe/+/859254 | 10:03 |
mnasiadka | hrw: I think yoctozepto promised, but if somebody else has time - would be nice to get it merged | 10:08 |
opendevreview | Marcin Juszkiewicz proposed openstack/kolla-ansible master: ansible-lint: cleanup our skip list https://review.opendev.org/c/openstack/kolla-ansible/+/859252 | 10:18 |
hrw | dropped baremetal comment | 10:18 |
kevko | Hi koalas ! | 11:20 |
opendevreview | Mathias Fechner proposed openstack/kolla-ansible master: Add ovn-exporter https://review.opendev.org/c/openstack/kolla-ansible/+/855498 | 11:33 |
hrw | kevko: https://review.opendev.org/c/openstack/kolla-ansible/+/859252 https://review.opendev.org/c/openstack/kolla/+/836664 wait ;D | 11:41 |
kevko | hrw: going to check, please ..could you share your opinion for this ? https://review.opendev.org/c/openstack/kolla-ansible/+/856906 (there is -1 for zull ..but it will be only something tiny bug...locally working) | 11:42 |
hrw | will look in a moment | 11:43 |
kevko | thanks | 11:45 |
opendevreview | Radosław Piliszek proposed openstack/kolla-ansible master: [CI] Change the Zun scenario to altcomp scenario https://review.opendev.org/c/openstack/kolla-ansible/+/851204 | 11:45 |
opendevreview | Radosław Piliszek proposed openstack/kolla-ansible master: Add Blazar to altcomp scenario https://review.opendev.org/c/openstack/kolla-ansible/+/853005 | 11:45 |
hrw | kevko: heh. why k-a just not have ansible in requirements.txt ;d | 11:48 |
kevko | hrw: i was asking also same question ! :D | 11:49 |
hrw | (kolla) 13:48 marcin@puchatek:kolla-ansible$ LANGUAGE=C kolla-ansible --help | 11:49 |
hrw | /home/marcin/.virtualenvs/kolla/bin/kolla-ansible: line 47: [: /usr/bin/python3: binary operator expected | 11:49 |
hrw | ERROR: kolla_ansible has to be available in the Ansible PYTHONPATH. | 11:49 |
hrw | Please install both in the same (virtual) environment. | 11:49 |
kevko | yoctozepto said that ansible is not official requirement of openstack/opendev .. so it is not in global requirements | 11:49 |
yoctozepto | but we can include it, it's just a legacy | 11:50 |
opendevreview | Radosław Piliszek proposed openstack/kolla-ansible master: Get OVS assignment right https://review.opendev.org/c/openstack/kolla-ansible/+/840895 | 11:50 |
mnasiadka | yoctozepto: something for the PTG? | 11:50 |
hrw | kevko: ansible_python_cmdline="/usr/bin/python3 -P" so your if fails due to lack of "" | 11:50 |
yoctozepto | mnasiadka: not sure, we can brainstorm during the regular meeting if there is a valid reason to omit ansible from the reqs when k-a is not really usable without it | 11:51 |
mnasiadka | we can, I'm fine with adding ansible-core to reqs | 11:52 |
mnasiadka | but I think global reqs will barf | 11:52 |
yoctozepto | yeah, especially ansible-core | 11:52 |
kevko | please, let's do it :) | 11:52 |
yoctozepto | we can disable barfing | 11:52 |
hrw | kevko: your patch should allow "kolla-ansible --help" | 11:53 |
kevko | because I am always confused when I am parsing kolla-ansible and count | 11:53 |
opendevreview | Radosław Piliszek proposed openstack/kolla-ansible master: Remove deprecated sysctl knobs https://review.opendev.org/c/openstack/kolla-ansible/+/855259 | 11:54 |
kevko | ansible-core or base and then ansible community version ...etc | 11:54 |
yoctozepto | anyone to review my long-standing patches..: https://review.opendev.org/q/project:openstack/kolla-ansible+status:open+branch:master+owner:radoslaw.piliszek%2540gmail.com+is:mergeable | 11:55 |
hrw | kevko: -1 for your patch | 11:55 |
yoctozepto | some are freshly renewed due to merge conflicts | 11:55 |
yoctozepto | what are the priorities now | 11:56 |
yoctozepto | I guess kolla centos 9? | 11:57 |
yoctozepto | and then systemd | 11:57 |
mnasiadka | yoctozepto: if you'll review Kolla Stream9 patch, then I can review yours :D | 11:58 |
opendevreview | Merged openstack/kolla-ansible master: ansible-lint: cleanup our skip list https://review.opendev.org/c/openstack/kolla-ansible/+/859252 | 11:59 |
yoctozepto | mnasiadka: well, it's unfair because mine have been there ready for longer :-) (also, there is one CR+2 that you can just restore because it was lost due to merge conflict resolution) | 11:59 |
opendevreview | Dr. Jens Harbott proposed openstack/kolla-ansible master: DNM: Failing a job to hold a node https://review.opendev.org/c/openstack/kolla-ansible/+/859214 | 11:59 |
yoctozepto | frickler: what are we debugging these days? ;-) | 12:00 |
mnasiadka | yoctozepto: one repository on quay.io was private, the ovn exporter:D | 12:00 |
mnasiadka | wonder if that's some changed default on quay.io side when you push to a nonexistent repo | 12:00 |
frickler | yoctozepto: ovn-exporter. I tried to set up the CI env locally, but after a day decided holding nodes is easier ... | 12:01 |
yoctozepto | frickler: ack | 12:01 |
yoctozepto | what's wrong with it? | 12:01 |
yoctozepto | mnasiadka: aye, that's the behaviour | 12:01 |
yoctozepto | I was fixing it until my script stopped working due to api changes | 12:01 |
yoctozepto | I agree the best solution is to ensure it's public from inside of the CI pipeline | 12:02 |
mnasiadka | I'll try to think of something Ansible-ish | 12:02 |
yoctozepto | ok, thanks | 12:03 |
hrw | yoctozepto: +1/+2 on part | 12:04 |
* hrw -> meeting | 12:04 | |
yoctozepto | hrw: thanks, appreciated | 12:07 |
yoctozepto | new link: | 12:09 |
yoctozepto | https://review.opendev.org/q/(projects:openstack/kolla+OR+project:openstack/ansible-collection-kolla)+status:open+branch:master+owner:radoslaw.piliszek%2540gmail.com+is:mergeable | 12:09 |
yoctozepto | added kolla and ansible-collection-kolla | 12:09 |
yoctozepto | (1 change per each but simple) | 12:09 |
dswebb | hi, quick one to see if any of you have seen this error before, using 14.3.0 (and 14.2.0-2) while trying to deploy ovn we get a templating error on our control nodes for the tunnel_interface which doesn't exist there https://pastebin.com/UxtvE1vm | 12:12 |
dswebb | weirdly enough we upgraded the cluster fine to yoga without hitting this error using largely the same globals we we're using now (only difference is tags for the ovn containers) | 12:13 |
kevko | btw, there was a question about stop one rabbitmq in cluster services stop working ... i think this is because of drop ha-all policy in rabbitmq role | 12:13 |
kevko | i had no time to verify it ...but i think this will be issue | 12:13 |
dswebb | we've confirmed that 100% the control nodes aren't part of the ovn-controller group so it has to be something with how the controllers are being interpreted for group_vars is our assumption | 12:16 |
Fl1nt | Good afternoon everybody! | 12:18 |
Fl1nt | Are there reasons out of "Someone should take care of it" for our haproxy version used on kolla/kolla-ansible is still using 1.8 line instead of the NFV 2.3/2.21 release (at least on CentOS)? | 12:20 |
Fl1nt | Using 2.x release, out of multiple performance and bug/security improvements would indeed also bring us the ability to not require the external prometheus exporter anymore ^^ | 12:20 |
yoctozepto | Fl1nt: I bet we have something newer with centos 9 | 12:23 |
Fl1nt | probably yeah, I was just looking at wallaby/xena/yoga and zed so was surprised to not see any 2.x release on HAPROXY | 12:31 |
Fl1nt | Oh btw, kinda related question, if we only build from source at some point, does that strategy also apply for satellite software like haproxy/elastic etc? | 12:34 |
yoctozepto | Fl1nt: no, it applies to openstack projects and it was a big big misnomer because, in fact, the openstack projects are always "from source", only the source of that source is different (distro packages [sub]projects vs now using upstream only) | 12:37 |
Fl1nt | yes I know about the weird naming, but what is kinda worrying with upstream sources build, is that offline platform (like our), will need to sync the whole opendev repo (almost) to be able to build internally and replace all URLs. | 12:39 |
Fl1nt | yoctozepto, Let say I would like to propose and contribute working on a way for offline based operators to specify two source of repository, what kind of formalism would you like? I can just contribute a bunch of patches here and there or have a more consistent/formal approach. | 12:43 |
yoctozepto | hrw: done https://review.opendev.org/c/openstack/kolla/+/836664 | 12:47 |
yoctozepto | Fl1nt: we use tarballs and it's far from "the whole opendev repo" ;-) | 12:48 |
yoctozepto | I think there were others interesting in offline work | 12:48 |
yoctozepto | maybe start a thread on the mailing list? | 12:48 |
yoctozepto | and pop up in the regular meetings and PTG to discuss it? | 12:48 |
frickler | why would you need to build images offline? can't you just build images and then publish those to your offline site? much easier than publishing the sources offline and doing major tweaking to the build process. not that overriding tarball sources should be much of an issue | 12:54 |
mnasiadka | Fl1nt: we used haproxy from NFV repo in some release | 12:59 |
mnasiadka | but I don't remember which | 13:00 |
rockey | frickler: for some users, it's easier to collect tarballs then building offline, than building online and converting to offline | 13:01 |
opendevreview | Martin Hiner proposed openstack/kolla-ansible master: Add systemd container control https://review.opendev.org/c/openstack/kolla-ansible/+/816724 | 13:05 |
hrw | Fl1nt: INFO:kolla.common.utils.haproxy: haproxy x86_64 2.4.17-3.el9 appstream 2.2 M | 13:06 |
hrw | Fl1nt: new enough? | 13:06 |
hrw | yoctozepto: thx | 13:07 |
Fl1nt | frickler, we can't, this is a highly locked up environment. | 13:08 |
Fl1nt | hrw, perfect! | 13:08 |
Fl1nt | hrw, seems cbs is lying to me ^^ | 13:09 |
hrw | Fl1nt: centos has 3 koji systems... | 13:09 |
Fl1nt | what do you mean? | 13:09 |
hrw | Fl1nt: https://kojihub.stream.centos.org/koji/buildinfo?buildID=23929 | 13:10 |
hrw | Fl1nt: ever used --tarballs-base argument to kolla-build? | 13:10 |
Fl1nt | oooh ok, bright new URL, ok I see thx. It's hard to keep up with all the streams moves ^^ | 13:11 |
hrw | Fl1nt: you just need to fetch tarballs online, mirror repos and then you can go offline | 13:11 |
hrw | Fl1nt: 3 kojis... sucks | 13:11 |
Fl1nt | Regarding tarballs it's fine, it's just that from time to time, on exporters and similar I should probably patch the whole dockerfiles and change ENV to ARG plus VERSION/URL variable format as for Openstack_exporter. | 13:12 |
hrw | Fl1nt: --template-override is another argument? | 13:13 |
hrw | ;D | 13:13 |
opendevreview | Verification of a change to openstack/kolla master failed: Don't ignore the remaining apt-get update errors https://review.opendev.org/c/openstack/kolla/+/856970 | 13:13 |
opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Change clouds.yaml location https://review.opendev.org/c/openstack/kolla-ansible/+/859056 | 13:14 |
Fl1nt | I'm willing to make it shorter, hence why I talked about having a more high precedence variable such as tarballs but for satellite artifacts, I know about template-override as offline install can't work without it ^^ | 13:14 |
hrw | Fl1nt: there is also --docker-dir | 13:15 |
Fl1nt | hrw, hold on you're using stream 9 :D You're damn to advanced in futur! we're still busy using c8s for now ^^ | 13:16 |
hrw | Fl1nt: c8s is past... | 13:16 |
opendevreview | Merged openstack/kolla-ansible master: [CI] Do not use the kolla user https://review.opendev.org/c/openstack/kolla-ansible/+/855017 | 13:18 |
Fl1nt | hrw, in here docker-dir doesn't matter, my artifacts are located on a local repository built for this environments specification (OS/Location/environment/etc) the whole build system create isolated but data optimized repositories (no data duplication unless isolation level required) | 13:18 |
Fl1nt | hrw, A new OS major version every 2 years and new packages everyday what a wonderful idea. | 13:19 |
hrw | Fl1nt: you may consider cs8 -> rockylinux9 then. | 13:21 |
Fl1nt | We already started a migration process of CentOS based to Debian based but it's long. | 13:22 |
opendevreview | Marcin Juszkiewicz proposed openstack/kolla master: nova-libvirt: provide TPM emulation https://review.opendev.org/c/openstack/kolla/+/857883 | 13:26 |
opendevreview | Marcin Juszkiewicz proposed openstack/kolla master: nova-libvirt: provide TPM emulation https://review.opendev.org/c/openstack/kolla/+/857883 | 13:26 |
Fl1nt | hum... just looking at patch list, is there any reason for this: https://review.opendev.org/c/openstack/kolla-ansible/+/816724 | 13:31 |
mnasiadka | the usual reason, podman | 13:33 |
Fl1nt | I know that CEPH-ansible is doing it that way but I'm wondering why as ansible docker modules can do anything docker related. Build/Run etc | 13:33 |
Fl1nt | mnasiadka, ah! yeah, ok. | 13:33 |
mnasiadka | Fl1nt: both ceph-ansible and cephadm do it this way, so you can plug in podman and docker into it ;) | 13:34 |
Fl1nt | right I keep forgetting about podman existing. | 13:34 |
Fl1nt | even if podman approach is kinda cool tbh | 13:34 |
Fl1nt | actually, systemd controling runC would probably be enough in the end. | 13:35 |
Fl1nt | BTW, regarding grafana, what's best, using template override to add this missing plugin (quite useful for alarming tbh) or to rework the grafana_plugin block to take care of random additional plugins? https://grafana.com/grafana/plugins/grafana-image-renderer/ | 13:44 |
Fl1nt | Grafana plugins are available for DEB/RPM based directly from their repo tbh | 13:44 |
Fl1nt | they can be installed from archives too. | 13:45 |
Fl1nt | not sure about that, I do have a downstream patch that use packages from rpm/deb repos on the grafana_plugin block if you want. | 13:46 |
Fl1nt | oh actually the patch do both ^^ | 13:46 |
hrw | Debian images are broken due to broken qemu upload | 14:37 |
opendevreview | Merged openstack/kolla-ansible master: Remove deprecated sysctl knobs https://review.opendev.org/c/openstack/kolla-ansible/+/855259 | 14:38 |
hrw | bifrost-- | 15:02 |
aidan | hello! I am running into an issue where the kolla-ansible commands with --limit are failing on the gen_facts step because one of the other hosts is down. Is there a way to configure it to not fail if another host is temporarily down? | 15:15 |
mnasiadka | yoctozepto: seems clouds.yaml followup is green now - care to reapply +2 https://review.opendev.org/c/openstack/kolla-ansible/+/859056 ? | 16:04 |
opendevreview | Alex Welsh proposed openstack/kayobe master: added support for ANSIBLE_VAULT_PASSWORD_FILE https://review.opendev.org/c/openstack/kayobe/+/858703 | 16:30 |
opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Specify number of threads for designate bind9 backend https://review.opendev.org/c/openstack/kolla-ansible/+/859325 | 17:41 |
yoctozepto | hrw: what do you mean by "Debian images are broken due to broken qemu upload"? | 17:46 |
yoctozepto | mnasiadka: ok | 17:46 |
hrw | yoctozepto: https://bugs.debian.org/1020776 | 17:47 |
yoctozepto | hrw: ouch, so building is broken? | 17:48 |
yoctozepto | arar | 17:48 |
hrw | yes | 17:48 |
yoctozepto | arar | 17:49 |
opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Specify number of threads for designate bind9 backend https://review.opendev.org/c/openstack/kolla-ansible/+/859325 | 17:57 |
kevko | yoctozepto, hrw ^^ https://review.opendev.org/c/openstack/kolla-ansible/+/859325 | 18:08 |
hrw | for me it probably needs releasenote | 18:10 |
kevko | hrw: understood | 18:12 |
hrw | maybe a way to specify >32 if someone has such need | 18:12 |
kevko | hrw: well, check the bug report :) ...that's the point ..it is not working for more than 32 workers :P | 18:13 |
kevko | hrw: and it is working out of the box ... if you want to specify some number ..you will just take the var and override in globals.yml ..that's it | 18:13 |
hrw | ok | 18:14 |
opendevreview | Radosław Piliszek proposed openstack/kolla-ansible master: Make Keystone admin endpoint creation optional https://review.opendev.org/c/openstack/kolla-ansible/+/843730 | 18:17 |
opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Specify number of threads for designate bind9 backend https://review.opendev.org/c/openstack/kolla-ansible/+/859325 | 18:18 |
kevko | hrw: reno added | 18:18 |
opendevreview | Marcin Juszkiewicz proposed openstack/kolla master: Move to CentOS Stream 9 https://review.opendev.org/c/openstack/kolla/+/836664 | 18:43 |
opendevreview | Marcin Juszkiewicz proposed openstack/kolla master: nova-libvirt: provide TPM emulation https://review.opendev.org/c/openstack/kolla/+/857883 | 18:43 |
opendevreview | Radosław Piliszek proposed openstack/kolla-ansible master: Add systemd container control https://review.opendev.org/c/openstack/kolla-ansible/+/816724 | 19:11 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!