Monday, 2022-09-26

opendevreviewDr. Jens Harbott proposed openstack/kolla-ansible master: DNM: Failing a job to hold a node  https://review.opendev.org/c/openstack/kolla-ansible/+/85921406:24
mnasiadkamorning07:26
fricklerdo we need to do something special in infra to set up new container images? docker.io/kolla/prometheus-ovn-exporter:master-ubuntu-jammy exists and I can pull it, but mirror.iad3.inmotion.opendev.org:4447/openstack.kolla/prometheus-ovn-exporter:master-ubuntu-jammy says "not authorized"07:48
mnasiadkawell, this is just a proxy in Apache, right?07:48
fricklerhmm, is it? /me goes to check logs07:49
mnasiadkaand we are rather going to quay.io?07:49
mnasiadkahttps://quay.io/repository/openstack.kolla/prometheus-ovn-exporter?tab=tags&tag=latest07:50
mnasiadkabut it's there, 4 hours ago07:50
fricklerah, o.k., the pull for that gives "not authorized" to me, too. the same for e.g. mysqld exporter works07:51
fricklerseems I need a Redhat login in order to be able to view that page?07:53
mnasiadkaah, that's interesting07:56
mnasiadkafor some reason the repo was marked as private07:57
fricklerright, https://quay.io/repository/openstack.kolla/prometheus-mysqld-exporter?tab=tags&tag=latest works without login07:57
mnasiadkaI changed it to public07:58
mnasiadkabut it seems we have 7 private repos07:58
mnasiadkalet me check07:58
fricklerthe ovn-exporter pull is working now, thx mnasiadka 08:00
mnasiadkaok, getting permissions of all repos, might take a while (need to do it via API)08:01
mnasiadkayoctozepto: seems new repositories created by docker push are getting marked as private - I think we need to extend the publish playbooks to ensure repos are public...08:03
priteauGood morning. I think ansible-lint released on Friday broke kolla-ansible CI08:07
priteauhttps://2ee6fa08daa1e3595342-a6ec19ba3cffa389cb7fa75d69a8d9a0.ssl.cf2.rackcdn.com/859113/2/check/openstack-tox-linters/be728b9/job-output.txt08:07
priteauname: Jinja templates should only be at the end of 'name' (name[template])08:08
priteauwhy oh why08:08
priteauDiscovered in https://review.opendev.org/c/openstack/kolla-ansible/+/859113 which broke between PS1 and PS2 hours apart :(08:09
mnasiadkadoesn't seem like hard to fix though ;-)08:09
priteauThis is the docs about it: https://ansible-lint.readthedocs.io/rules/name/08:10
priteauname[template] - Jinja templates should only be at the end of ‘name’. This helps with the identification of tasks inside the source code when they fail. The use of templating inside name keys is discouraged as there are multiple cases where the rendering of the name template is not possible.08:10
priteaumnasiadka: Are you ok with adding this rule to skip_list?08:10
priteauI don't think we have issues with rendering of the name template08:11
mnasiadkaIf it takes days to fix, then yes08:11
mnasiadkabut I assume we do loop on templating08:11
priteauIt fails on things like this:08:14
priteau- name: Check if the folder ["{{ node_custom_config }}/ceilometer/{{ ceilometer_dynamic_pollsters_local_folder }}"] for dynamic pollsters definitions exist08:14
priteauI really don't see how putting it at the end of the name field makes it better?08:15
priteauI see, because it would truncate anything after08:16
priteauNot a big issue really08:16
mnasiadkawell, I'm not a fan of adding things to skip_list forever, we already do have quite a collection of skip_list - if it's easy to fix it, then we should just fix it08:18
priteauWe must not bow to the ansible-lint dictatorship!08:19
priteauOr we will do a follow up08:20
opendevreviewPierre Riteau proposed openstack/kolla-ansible master: CI: allow Jinja templating anywhere inside task and play names  https://review.opendev.org/c/openstack/kolla-ansible/+/85922008:20
mnasiadkafrickler: seems it was the only private repository08:25
priteauCI checks passed ^^^08:30
opendevreviewMerged openstack/kolla-ansible master: CI: allow Jinja templating anywhere inside task and play names  https://review.opendev.org/c/openstack/kolla-ansible/+/85922008:48
opendevreviewPierre Riteau proposed openstack/kolla-ansible master: Fix Ironic API healthcheck with backend TLS  https://review.opendev.org/c/openstack/kolla-ansible/+/85922208:52
opendevreviewPierre Riteau proposed openstack/kolla-ansible master: Fix interface option for ironic-neutron-agent  https://review.opendev.org/c/openstack/kolla-ansible/+/85911308:52
opendevreviewIvan Halomi proposed openstack/kolla-ansible master: Add support of podman deployment  https://review.opendev.org/c/openstack/kolla-ansible/+/79922909:04
opendevreviewMichal Nasiadka proposed openstack/kolla master: Add Opensearch image(s)  https://review.opendev.org/c/openstack/kolla/+/83037309:15
opendevreviewVerification of a change to openstack/kayobe master failed: Update to latest CentOS Stream 8 cloud image  https://review.opendev.org/c/openstack/kayobe/+/85786909:22
opendevreviewMichal Nasiadka proposed openstack/kolla master: Add Opensearch image(s)  https://review.opendev.org/c/openstack/kolla/+/83037309:32
opendevreviewMarcin Juszkiewicz proposed openstack/kolla-ansible master: ansible-lint: cleanup our skip list  https://review.opendev.org/c/openstack/kolla-ansible/+/85925209:44
hrwas we do ansible-lint cleanups...09:44
mnasiadkahrw: we don't have baremetal role anymore, so maybe we don't need all of them ;)09:46
hrwyou are too optimistic ;D09:46
hrwFinished with 1676 failure(s), 9 warning(s) on 1774 files. (without .ansible-lint)09:47
hrwok. who will review centos 9 patch? https://review.opendev.org/c/openstack/kolla/+/83666409:49
hrwas it feels abandoned09:50
admin1i added a component ( octavia ) in an existing cluster and the playbooks completed . but horizon shows internal server error .. how do I troubleshoot it ? 09:59
opendevreviewPierre Riteau proposed openstack/kayobe master: [release] Set previous_release to yoga  https://review.opendev.org/c/openstack/kayobe/+/85925410:03
mnasiadkahrw: I think yoctozepto promised, but if somebody else has time - would be nice to get it merged10:08
opendevreviewMarcin Juszkiewicz proposed openstack/kolla-ansible master: ansible-lint: cleanup our skip list  https://review.opendev.org/c/openstack/kolla-ansible/+/85925210:18
hrwdropped baremetal comment10:18
kevkoHi koalas ! 11:20
opendevreviewMathias Fechner proposed openstack/kolla-ansible master: Add ovn-exporter  https://review.opendev.org/c/openstack/kolla-ansible/+/85549811:33
hrwkevko: https://review.opendev.org/c/openstack/kolla-ansible/+/859252 https://review.opendev.org/c/openstack/kolla/+/836664 wait ;D11:41
kevkohrw: going to check, please ..could you share your opinion for this ? https://review.opendev.org/c/openstack/kolla-ansible/+/856906 (there is -1 for zull ..but it will be only something tiny bug...locally working)11:42
hrwwill look in a moment11:43
kevkothanks11:45
opendevreviewRadosław Piliszek proposed openstack/kolla-ansible master: [CI] Change the Zun scenario to altcomp scenario  https://review.opendev.org/c/openstack/kolla-ansible/+/85120411:45
opendevreviewRadosław Piliszek proposed openstack/kolla-ansible master: Add Blazar to altcomp scenario  https://review.opendev.org/c/openstack/kolla-ansible/+/85300511:45
hrwkevko: heh. why k-a just not have ansible in requirements.txt ;d11:48
kevkohrw: i was asking also same question ! :D 11:49
hrw(kolla) 13:48 marcin@puchatek:kolla-ansible$ LANGUAGE=C kolla-ansible --help11:49
hrw/home/marcin/.virtualenvs/kolla/bin/kolla-ansible: line 47: [: /usr/bin/python3: binary operator expected11:49
hrwERROR: kolla_ansible has to be available in the Ansible PYTHONPATH.11:49
hrwPlease install both in the same (virtual) environment.11:49
kevkoyoctozepto said that ansible is not official requirement of openstack/opendev .. so it is not in global requirements 11:49
yoctozeptobut we can include it, it's just a legacy11:50
opendevreviewRadosław Piliszek proposed openstack/kolla-ansible master: Get OVS assignment right  https://review.opendev.org/c/openstack/kolla-ansible/+/84089511:50
mnasiadkayoctozepto: something for the PTG?11:50
hrwkevko: ansible_python_cmdline="/usr/bin/python3 -P" so your if fails due to lack of ""11:50
yoctozeptomnasiadka: not sure, we can brainstorm during the regular meeting if there is a valid reason to omit ansible from the reqs when k-a is not really usable without it11:51
mnasiadkawe can, I'm fine with adding ansible-core to reqs11:52
mnasiadkabut I think global reqs will barf11:52
yoctozeptoyeah, especially ansible-core11:52
kevkoplease, let's do it :)11:52
yoctozeptowe can disable barfing11:52
hrwkevko: your patch should allow "kolla-ansible --help"11:53
kevkobecause I am always confused when I am parsing kolla-ansible and count11:53
opendevreviewRadosław Piliszek proposed openstack/kolla-ansible master: Remove deprecated sysctl knobs  https://review.opendev.org/c/openstack/kolla-ansible/+/85525911:54
kevkoansible-core or base and then ansible community version ...etc11:54
yoctozeptoanyone to review my long-standing patches..: https://review.opendev.org/q/project:openstack/kolla-ansible+status:open+branch:master+owner:radoslaw.piliszek%2540gmail.com+is:mergeable11:55
hrwkevko: -1 for your patch11:55
yoctozeptosome are freshly renewed due to merge conflicts11:55
yoctozeptowhat are the priorities now11:56
yoctozeptoI guess kolla centos 9?11:57
yoctozeptoand then systemd11:57
mnasiadkayoctozepto: if you'll review Kolla Stream9 patch, then I can review yours :D11:58
opendevreviewMerged openstack/kolla-ansible master: ansible-lint: cleanup our skip list  https://review.opendev.org/c/openstack/kolla-ansible/+/85925211:59
yoctozeptomnasiadka: well, it's unfair because mine have been there ready for longer :-) (also, there is one CR+2 that you can just restore because it was lost due to merge conflict resolution)11:59
opendevreviewDr. Jens Harbott proposed openstack/kolla-ansible master: DNM: Failing a job to hold a node  https://review.opendev.org/c/openstack/kolla-ansible/+/85921411:59
yoctozeptofrickler: what are we debugging these days? ;-)12:00
mnasiadkayoctozepto: one repository on quay.io was private, the ovn exporter:D12:00
mnasiadkawonder if that's some changed default on quay.io side when you push to a nonexistent repo12:00
frickleryoctozepto: ovn-exporter. I tried to set up the CI env locally, but after a day decided holding nodes is easier ...12:01
yoctozeptofrickler: ack12:01
yoctozeptowhat's wrong with it?12:01
yoctozeptomnasiadka: aye, that's the behaviour12:01
yoctozeptoI was fixing it until my script stopped working due to api changes12:01
yoctozeptoI agree the best solution is to ensure it's public from inside of the CI pipeline12:02
mnasiadkaI'll try to think of something Ansible-ish12:02
yoctozeptook, thanks12:03
hrwyoctozepto: +1/+2 on part 12:04
* hrw -> meeting12:04
yoctozeptohrw: thanks, appreciated12:07
yoctozeptonew link:12:09
yoctozeptohttps://review.opendev.org/q/(projects:openstack/kolla+OR+project:openstack/ansible-collection-kolla)+status:open+branch:master+owner:radoslaw.piliszek%2540gmail.com+is:mergeable12:09
yoctozeptoadded kolla and ansible-collection-kolla12:09
yoctozepto(1 change per each but simple)12:09
dswebbhi, quick one to see if any of you have seen this error before, using 14.3.0 (and 14.2.0-2) while trying to deploy ovn we get a templating error on our control nodes for the tunnel_interface which doesn't exist there https://pastebin.com/UxtvE1vm12:12
dswebbweirdly enough we upgraded the cluster fine to yoga without hitting this error using largely the same globals we we're using now (only difference is tags for the ovn containers)12:13
kevkobtw, there was a question about stop one rabbitmq in cluster services stop working ... i think this is because of drop ha-all policy in rabbitmq role 12:13
kevkoi had no time to verify it ...but i think this will be issue 12:13
dswebbwe've confirmed that 100% the control nodes aren't part of the ovn-controller group so it has to be something with how the controllers are being interpreted for group_vars is our assumption12:16
Fl1ntGood afternoon everybody!12:18
Fl1ntAre there reasons out of "Someone should take care of it" for our haproxy version used on kolla/kolla-ansible is still using 1.8 line instead of the NFV 2.3/2.21 release (at least on CentOS)?12:20
Fl1ntUsing 2.x release, out of multiple performance and bug/security improvements would indeed also bring us the ability to not require the external prometheus exporter anymore ^^12:20
yoctozeptoFl1nt: I bet we have something newer with centos 912:23
Fl1ntprobably yeah, I was just looking at wallaby/xena/yoga and zed so was surprised to not see any 2.x release on HAPROXY12:31
Fl1ntOh btw, kinda related question, if we only build from source at some point, does that strategy also apply for satellite software like haproxy/elastic etc?12:34
yoctozeptoFl1nt: no, it applies to openstack projects and it was a big big misnomer because, in fact, the openstack projects are always "from source", only the source of that source is different (distro packages [sub]projects vs now using upstream only)12:37
Fl1ntyes I know about the weird naming, but what is kinda worrying with upstream sources build, is that offline platform (like our), will need to sync the whole opendev repo (almost) to be able to build internally and replace all URLs.12:39
Fl1ntyoctozepto, Let say I would like to propose and contribute working on a way for offline based operators to specify two source of repository, what kind of formalism would you like? I can just contribute a bunch of patches here and there or have a more consistent/formal approach.12:43
yoctozeptohrw: done https://review.opendev.org/c/openstack/kolla/+/83666412:47
yoctozeptoFl1nt: we use tarballs and it's far from "the whole opendev repo" ;-)12:48
yoctozeptoI think there were others interesting in offline work12:48
yoctozeptomaybe start a thread on the mailing list?12:48
yoctozeptoand pop up in the regular meetings and PTG to discuss it?12:48
fricklerwhy would you need to build images offline? can't you just build images and then publish those to your offline site? much easier than publishing the sources offline and doing major tweaking to the build process. not that overriding tarball sources should be much of an issue12:54
mnasiadkaFl1nt: we used haproxy from NFV repo in some release12:59
mnasiadkabut I don't remember which13:00
rockeyfrickler: for some users, it's easier to collect tarballs then building offline, than building online and converting to offline13:01
opendevreviewMartin Hiner proposed openstack/kolla-ansible master: Add systemd container control  https://review.opendev.org/c/openstack/kolla-ansible/+/81672413:05
hrwFl1nt: INFO:kolla.common.utils.haproxy: haproxy         x86_64         2.4.17-3.el9            appstream         2.2 M13:06
hrwFl1nt: new enough?13:06
hrwyoctozepto: thx13:07
Fl1ntfrickler, we can't, this is a highly locked up environment.13:08
Fl1nthrw, perfect!13:08
Fl1nthrw, seems cbs is lying to me ^^13:09
hrwFl1nt: centos has 3 koji systems...13:09
Fl1ntwhat do you mean?13:09
hrwFl1nt: https://kojihub.stream.centos.org/koji/buildinfo?buildID=2392913:10
hrwFl1nt: ever used --tarballs-base argument to kolla-build?13:10
Fl1ntoooh ok, bright new URL, ok I see thx. It's hard to keep up with all the streams moves ^^13:11
hrwFl1nt: you just need to fetch tarballs online, mirror repos and then you can go offline13:11
hrwFl1nt: 3 kojis... sucks13:11
Fl1ntRegarding tarballs it's fine, it's just that from time to time, on exporters and similar I should probably patch the whole dockerfiles and change ENV to ARG plus VERSION/URL variable format as for Openstack_exporter.13:12
hrwFl1nt: --template-override is another argument?13:13
hrw;D13:13
opendevreviewVerification of a change to openstack/kolla master failed: Don't ignore the remaining apt-get update errors  https://review.opendev.org/c/openstack/kolla/+/85697013:13
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: Change clouds.yaml location  https://review.opendev.org/c/openstack/kolla-ansible/+/85905613:14
Fl1ntI'm willing to make it shorter, hence why I talked about having a more high precedence variable such as tarballs but for satellite artifacts, I know about template-override as offline install can't work without it ^^13:14
hrwFl1nt: there is also --docker-dir13:15
Fl1nthrw, hold on you're using stream 9 :D You're damn to advanced in futur! we're still busy using c8s for now ^^13:16
hrwFl1nt: c8s is past...13:16
opendevreviewMerged openstack/kolla-ansible master: [CI] Do not use the kolla user  https://review.opendev.org/c/openstack/kolla-ansible/+/85501713:18
Fl1nthrw, in here docker-dir doesn't matter, my artifacts are located on a local repository built for this environments specification (OS/Location/environment/etc) the whole build system create isolated but data optimized repositories (no data duplication unless isolation level required)13:18
Fl1nthrw, A new OS major version every 2 years and new packages everyday what a wonderful idea.13:19
hrwFl1nt: you may consider cs8 -> rockylinux9 then.13:21
Fl1ntWe already started a migration process of CentOS based to Debian based but it's long.13:22
opendevreviewMarcin Juszkiewicz proposed openstack/kolla master: nova-libvirt: provide TPM emulation  https://review.opendev.org/c/openstack/kolla/+/85788313:26
opendevreviewMarcin Juszkiewicz proposed openstack/kolla master: nova-libvirt: provide TPM emulation  https://review.opendev.org/c/openstack/kolla/+/85788313:26
Fl1nthum... just looking at patch list, is there any reason for this: https://review.opendev.org/c/openstack/kolla-ansible/+/81672413:31
mnasiadkathe usual reason, podman13:33
Fl1ntI know that CEPH-ansible is doing it that way but I'm wondering why as ansible docker modules can do anything docker related. Build/Run etc13:33
Fl1ntmnasiadka, ah! yeah, ok.13:33
mnasiadkaFl1nt: both ceph-ansible and cephadm do it this way, so you can plug in podman and docker into it ;)13:34
Fl1ntright I keep forgetting about podman existing.13:34
Fl1nteven if podman approach is kinda cool tbh13:34
Fl1ntactually, systemd controling runC would probably be enough in the end.13:35
Fl1ntBTW, regarding grafana, what's best, using template override to add this missing plugin (quite useful for alarming tbh) or to rework the grafana_plugin block to take care of random additional plugins? https://grafana.com/grafana/plugins/grafana-image-renderer/13:44
Fl1ntGrafana plugins are available for DEB/RPM based directly from their repo tbh13:44
Fl1ntthey can be installed from archives too.13:45
Fl1ntnot sure about that, I do have a downstream patch that use packages from rpm/deb repos on the grafana_plugin block if you want.13:46
Fl1ntoh actually the patch do both ^^13:46
hrwDebian images are broken due to broken qemu upload14:37
opendevreviewMerged openstack/kolla-ansible master: Remove deprecated sysctl knobs  https://review.opendev.org/c/openstack/kolla-ansible/+/85525914:38
hrwbifrost--15:02
aidanhello! I am running into an issue where the kolla-ansible commands with --limit are failing on the gen_facts step because one of the other hosts is down. Is there a way to configure it to not fail if another host is temporarily down? 15:15
mnasiadkayoctozepto: seems clouds.yaml followup is green now - care to reapply +2 https://review.opendev.org/c/openstack/kolla-ansible/+/859056 ?16:04
opendevreviewAlex Welsh proposed openstack/kayobe master: added support for ANSIBLE_VAULT_PASSWORD_FILE  https://review.opendev.org/c/openstack/kayobe/+/85870316:30
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Specify number of threads for designate bind9 backend  https://review.opendev.org/c/openstack/kolla-ansible/+/85932517:41
yoctozeptohrw: what do you mean by "Debian images are broken due to broken qemu upload"?17:46
yoctozeptomnasiadka: ok17:46
hrwyoctozepto: https://bugs.debian.org/102077617:47
yoctozeptohrw: ouch, so building is broken?17:48
yoctozeptoarar17:48
hrwyes17:48
yoctozeptoarar17:49
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Specify number of threads for designate bind9 backend  https://review.opendev.org/c/openstack/kolla-ansible/+/85932517:57
kevkoyoctozepto, hrw ^^ https://review.opendev.org/c/openstack/kolla-ansible/+/859325 18:08
hrwfor me it probably needs releasenote18:10
kevkohrw: understood 18:12
hrwmaybe a way to specify >32 if someone has such need18:12
kevkohrw: well, check the bug report :) ...that's the point ..it is not working for more than 32 workers :P 18:13
kevkohrw: and it is working out of the box ... if you want to specify some number ..you will just take the var and override in globals.yml ..that's it18:13
hrwok18:14
opendevreviewRadosław Piliszek proposed openstack/kolla-ansible master: Make Keystone admin endpoint creation optional  https://review.opendev.org/c/openstack/kolla-ansible/+/84373018:17
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Specify number of threads for designate bind9 backend  https://review.opendev.org/c/openstack/kolla-ansible/+/85932518:18
kevkohrw: reno added18:18
opendevreviewMarcin Juszkiewicz proposed openstack/kolla master: Move to CentOS Stream 9  https://review.opendev.org/c/openstack/kolla/+/83666418:43
opendevreviewMarcin Juszkiewicz proposed openstack/kolla master: nova-libvirt: provide TPM emulation  https://review.opendev.org/c/openstack/kolla/+/85788318:43
opendevreviewRadosław Piliszek proposed openstack/kolla-ansible master: Add systemd container control  https://review.opendev.org/c/openstack/kolla-ansible/+/81672419:11

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!