| jrwr | Running Latest Kolla, Using the public containers, Nova-libvirt doesn't contain OVMF causing nova not to be able to boot UEFI instances :( | 04:34 |
|---|---|---|
| jrwr | Odd, its in the container | 04:37 |
| jrwr | https://www.irccloud.com/pastebin/vMBtV9zr/ | 04:37 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/xena: Remove missing collectd packages https://review.opendev.org/c/openstack/kolla/+/827740 | 06:24 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/wallaby: Remove missing collectd packages https://review.opendev.org/c/openstack/kolla/+/827741 | 06:24 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/victoria: Drop CentOS Linux 8 image builds https://review.opendev.org/c/openstack/kolla/+/827608 | 06:35 |
| bjolo_ | why is the elasticsearc version so old in kolla? | 08:46 |
| bjolo_ | 7.10 vs latest 7.17 | 08:47 |
| kevko | hi, do you know why collectd for centos8stream is failing ..it seems some packages are missing in repo ? | 08:47 |
| hrw | bjolo_: old image? | 08:48 |
| hrw | kevko: yes, we know | 08:48 |
| hrw | 07:24 < opendevreview> Michal Nasiadka proposed openstack/kolla stable/xena: Remove missing collectd packages https://review.opendev.org/c/openstack/kolla/+/827740 | 08:48 |
| hrw | 07:24 < opendevreview> Michal Nasiadka proposed openstack/kolla stable/wallaby: Remove missing collectd packages https://review.opendev.org/c/openstack/kolla/+/827741 | 08:48 |
| kevko | hrw: can I find somewhere some proposed change I can rebase onto to pass zuul ? | 08:48 |
| hrw | not checked today's CI queue yet | 08:49 |
| mnasiadka | kevko: master fix has been merged | 08:49 |
| kevko | aa, thanks | 08:49 |
| mnasiadka | kevko: https://review.opendev.org/c/openstack/kolla/+/827679 | 08:50 |
| opendevreview | Marcin Juszkiewicz proposed openstack/kolla master: Switch to use Yoga binary packages https://review.opendev.org/c/openstack/kolla/+/826488 | 08:50 |
| hrw | kevko: so just press 'rebase' button in gerrit | 08:50 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/wallaby: Remove missing collectd packages https://review.opendev.org/c/openstack/kolla/+/827741 | 09:02 |
| mnasiadka | ok, Wallaby should pass now as well (Nautilus repo needs to be from RDO for now) | 09:03 |
| frickler | bjolo_: eleasticsearch stopped being opensource (depending on your definition), so we don't use the newer versions | 09:11 |
| hrw | kevko, mnasiadka: https://review.opendev.org/c/openstack/kolla/+/826488 UCA/RDO for R-8 waits for you | 09:11 |
| * kevko going to check | 09:12 | |
| opendevreview | Marcin Juszkiewicz proposed openstack/kolla master: erlang: use packages from Erlang Solutions on AArch64 https://review.opendev.org/c/openstack/kolla/+/827663 | 09:15 |
| kevko | hrw: +2, looks good let's wait for zuul | 09:16 |
| opendevreview | Marcin Juszkiewicz proposed openstack/kolla master: Revert "nova: drop pypowervm dependency" https://review.opendev.org/c/openstack/kolla/+/827240 | 09:16 |
| opendevreview | Michal Arbet proposed openstack/kolla master: Add mariadb-arbitrator image https://review.opendev.org/c/openstack/kolla/+/780967 | 09:17 |
| hrw | kevko: note that zuul can give +1 on that patch even when it is not working | 09:18 |
| kevko | hrw: i know :) | 09:18 |
| hrw | as binary jobs are non-voting | 09:18 |
| kevko | i am also switching to source | 09:19 |
| kevko | I'm looking forward to deleting the binary scope ... it will be more clean :P | 09:21 |
| kevko | btw, can I switch downloading of service sources from git instead of tarballs ? Hope you will be faster to reply than me looking into code | 09:23 |
| hrw | yes, you can. I just do not remember how | 09:24 |
| hrw | kevko: tarballs are generated after each commit so they are up-to-date | 09:24 |
| kevko | yeah, i know, but sometimes when i am debugging something ..it's confortable to git log in a container :) im just curious | 09:25 |
| frickler | yoctozepto: we have a set of nodes held now after your recheck yesterday, let me know if you want access or if I should simply drop them again | 09:25 |
| hrw | kevko: look into kolla/common/config.py - SOURCES dict. most are type:url but some are type:git - you need to change entries | 09:25 |
| hrw | kevko: what you need is so called 'kolla_dev_mode' | 09:26 |
| kevko | ah, ok | 09:26 |
| kevko | thank you | 09:26 |
| hrw | kevko: not all containers have it and it is on k-a side | 09:26 |
| hrw | kevko: source is outside of image and is bind mounted | 09:26 |
| opendevreview | Konstantin Yarovoy proposed openstack/kolla master: Refactor image module and transition to DockerClient https://review.opendev.org/c/openstack/kolla/+/825786 | 09:29 |
| mnasiadka | hrw, kevko: still I think with the transition to source, it would be good to document how users can find out the last commit sha - date might not be enough in some cases. Probably there should be something in egg data (git commit hash maybe?) | 09:42 |
| opendevreview | Konstantin Yarovoy proposed openstack/kolla master: Add podman for image building https://review.opendev.org/c/openstack/kolla/+/825791 | 09:43 |
| hrw | mnasiadka: kind of what kevko wants in https://review.opendev.org/c/openstack/kolla/+/818727 one? | 09:43 |
| mnasiadka | hrw: I'm rather referring to e.g. Neutron source git commit hash, not Kolla's ;-) | 09:44 |
| hrw | ah, no - it just adds kolla git | 09:44 |
| hrw | mnasiadka: we use tarballs so there is no git info | 09:44 |
| hrw | and imho if you depend on some recent fix then you can always build own images. which we suggest anyway | 09:45 |
| kevko | yeah, that was also the reason why i wanted to use git .. | 09:45 |
| kevko | because i really now what is the version checkouted inside ..and also i can check changes with git show and maybe has some picture what was changed - if something is working in bad way .. | 09:46 |
| kevko | but, even if tarball is used ..build process can check last commit in branch on github and maybe set env variable ..so if i docker inspect ..i will see the commit | 09:47 |
| hrw | kevko: 1. github is mirror 2. you can get desync between fetching tarball and fresh commit | 09:49 |
| kevko | hrw: ah, ok so not github ..but opendev ? what is the primary git location ? | 09:49 |
| kevko | opendev ? | 09:49 |
| hrw | yes | 09:49 |
| hrw | OpenStack is opendev project so that's where it is stored | 09:50 |
| kevko | this would be quite easy ..just we have to have list of git locations (same as for tarballs) ...and then just get info about last commit and set as env ? | 09:50 |
| hrw | and you fetch tarball at 9:50:02 and there is commit on 9:50:03. you fetch tarball, check commit and get newer data than in tarball | 09:53 |
| mnasiadka | it would be easier if tarballs would have some metadata ;) | 09:53 |
| kevko | hrw: well, yes, that's true .. | 09:54 |
| kevko | hrw: but instead of tarballs webpage you can download tarball directly from git, can't you ? | 09:54 |
| hrw | kevko: infra team will hate you | 09:55 |
| hrw | ok, overreacted but it will be not wise way imho | 09:55 |
| kevko | hh, ok .. i said nothing :D | 09:56 |
| hrw | they made tarballs as a service for such needs | 09:56 |
| frickler | but the question of how to map a tarball to a git commit is an interesting one indeed, maybe you want to take that question to #opendev and discuss there | 09:57 |
| mnasiadka | frickler: done, let's see | 10:00 |
| hrw | Version: 13.0.2.dev5 in PKG-INFO you have in tarball (thats for xena) | 10:00 |
| mnasiadka | well, that's pbr's version, doesn't help a lot I guess | 10:01 |
| hrw | where 13.0.2dev5 means 5 commits after tag | 10:01 |
| hrw | still something | 10:01 |
| mnasiadka | hrw: I don't know if neutron really works for Ubuntu binary: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_330/826488/4/check/kolla-ansible-ubuntu-binary/330d33f/primary/logs/kolla/all-ERROR.txt | 10:01 |
| hrw | mnasiadka: I checked building. not k-a | 10:02 |
| mnasiadka | understood | 10:02 |
| mnasiadka | from another perspective, vm boots and is accessible via ssh | 10:02 |
| mnasiadka | so I guess that's main OpenStack use case ;) | 10:03 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/wallaby: Remove missing collectd packages https://review.opendev.org/c/openstack/kolla/+/827741 | 10:40 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/victoria: Drop CentOS Linux 8 image builds https://review.opendev.org/c/openstack/kolla/+/827608 | 10:42 |
| opendevreview | Marcin Juszkiewicz proposed openstack/kolla master: stringify docker-squash version https://review.opendev.org/c/openstack/kolla/+/827835 | 10:48 |
| opendevreview | Mark Goddard proposed openstack/kayobe stable/wallaby: Set correct gateway for the bifrost provision network https://review.opendev.org/c/openstack/kayobe/+/827744 | 10:56 |
| opendevreview | Mark Goddard proposed openstack/kayobe stable/victoria: Set correct gateway for the bifrost provision network https://review.opendev.org/c/openstack/kayobe/+/827745 | 10:56 |
| yoctozepto | frickler: I want to access them, pm me | 11:07 |
| opendevreview | Mark Goddard proposed openstack/kayobe master: DNM: enable iPXE for UEFI https://review.opendev.org/c/openstack/kayobe/+/827667 | 11:09 |
| opendevreview | Marcin Juszkiewicz proposed openstack/kolla master: rabbitmq/centos: update to 3.9 https://review.opendev.org/c/openstack/kolla/+/827656 | 11:23 |
| hrw | mnasiadka: rebased on top of aarch64 erlang update | 11:23 |
| mnasiadka | thanks | 11:23 |
| hrw | mnasiadka: Debian is still on 23/3.8 | 11:34 |
| opendevreview | Marcin Juszkiewicz proposed openstack/kolla master: pin out some package from Debian OpenStack Team repos https://review.opendev.org/c/openstack/kolla/+/827848 | 11:36 |
| hrw | I reported that several times. enough. | 11:36 |
| hrw | kevko: https://review.opendev.org/c/openstack/kolla/+/827848 | 11:42 |
| opendevreview | Marcin Juszkiewicz proposed openstack/kolla master: rabbitmq: update to 3.9 https://review.opendev.org/c/openstack/kolla/+/827656 | 11:44 |
| hrw | mnasiadka: Debian updated to 24/3.9 too | 11:44 |
| yoctozepto | hrw, mnasiadka, frickler, kevko: using git and its metadata is a wonderful idea; we could even stretch it to "kolla, show me what new commits I will get if I rebuild now" (assuming there was some cached metadata of the previous run) | 11:50 |
| yoctozepto | let's discuss this during the upcoming ptg for sure | 11:50 |
| opendevreview | Merged openstack/kayobe stable/xena: ntp: Fix service mask when service doesn't exist https://review.opendev.org/c/openstack/kayobe/+/827404 | 11:51 |
| opendevreview | Merged openstack/kolla stable/xena: Remove missing collectd packages https://review.opendev.org/c/openstack/kolla/+/827740 | 11:56 |
| opendevreview | Merged openstack/kolla master: Switch to use Yoga binary packages https://review.opendev.org/c/openstack/kolla/+/826488 | 11:56 |
| jingvar | Hi! | 11:58 |
| jingvar | How can I add nova available_filters ? | 11:59 |
| shyamb | Hi Have we created any install doc for centos stream 8 victoria? | 12:08 |
| shyamb | or updated the existing one? | 12:08 |
| shyamb | or it's in-progress | 12:08 |
| priteau | jingvar: do you need filters not shipped with nova itself? | 12:22 |
| jingvar | I just run tempest :) | 12:23 |
| jingvar | there are tests to boot VMs on diffrernt hosts | 12:23 |
| jingvar | it fails because samehost -differnt hosts are not enebled | 12:24 |
| jingvar | for example tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_cross_tenant_traffic | 12:25 |
| priteau | Isn't this done with server groups and anti-affinity? | 12:34 |
| jingvar | priteau: As I understand tempest uses different approach | 12:46 |
| priteau | Anyway, if you just want filters already in nova, you should be able to just add them to enabled_filters via nova.conf | 12:46 |
| priteau | available_filters is used to import out-of-tree filters | 12:47 |
| jingvar | https://docs.openstack.org/nova/queens/admin/configuration/schedulers.html#differenthostfilter | 12:47 |
| jingvar | but how customize it by Kayobe, whithout whole override nova.conf | 12:48 |
| priteau | you just add the [filter_scheduler] section and enabled_filters line to etc/kayobe/kolla/config/nova/nova-scheduler.conf | 12:58 |
| priteau | Kolla Ansible will merge with its defaults | 12:58 |
| mnasiadka | mgoddard: https://review.opendev.org/c/openstack/kolla/+/827741 - want to lift your -1 since now it's not CI only? ;-) | 12:59 |
| mgoddard | mnasiadka: done | 13:22 |
| mnasiadka | hrw, yoctozepto: got a minute for https://review.opendev.org/c/openstack/kolla/+/827741 ? | 13:23 |
| mnasiadka | mgoddard: thx | 13:23 |
| mnasiadka | the same goes for this - https://review.opendev.org/c/openstack/kolla/+/827608 | 13:23 |
| opendevreview | Merged openstack/kayobe master: ntp: Fix service mask when service doesn't exist https://review.opendev.org/c/openstack/kayobe/+/827486 | 13:37 |
| jingvar | priteau: thanks | 13:47 |
| opendevreview | Merged openstack/kolla stable/wallaby: Remove missing collectd packages https://review.opendev.org/c/openstack/kolla/+/827741 | 14:49 |
| opendevreview | Merged openstack/kolla-ansible stable/wallaby: Add OIDCDiscoverURL mod_oidc option https://review.opendev.org/c/openstack/kolla-ansible/+/827638 | 15:21 |
| hrw | dine | 15:29 |
| admiyo | hrw, thanks for the guidance yesterday with the "file not found config.json" issue. I've learned a little more, and I think it is worth sharing. I think what is happeneing is an earlier failure keeps the configs from being generated. I noticed that a Nova step is failing as well, and I think that is the culprit. If I pregenerate the configs and the n run a reeconfigure, the neutron bootstrap (and the rest of the deploy) completes, | 15:48 |
| admiyo | but there are no hypervisores, which leads me to think it is all driven from the Nova failure. | 15:48 |
| hrw | INFO:kolla.common.utils.collectd:[91mError: Unable to find a match: collectd-pcie-errors - looks like we have another one to drop (aarch64) | 16:10 |
| hrw | collectd-pcie-errors is now x86-64 only | 16:14 |
| opendevreview | Marcin Juszkiewicz proposed openstack/kolla master: collectd: pcie-errors is x86-64 only now https://review.opendev.org/c/openstack/kolla/+/827883 | 16:15 |
| hrw | mnasiadka, mgoddard, yoctozepto and other cores: we have some AArch64 CI fixes to review. https://review.opendev.org/c/openstack/kolla/+/827883 (collectd again), https://review.opendev.org/c/openstack/kolla/+/827663 (erlang so rabbitmq works), https://review.opendev.org/c/openstack/kolla/+/827848 (fixes Debian builds) | 16:18 |
| hrw | each got RP+2 | 16:19 |
| opendevreview | Marcin Juszkiewicz proposed openstack/kolla master: Require extra argument for building binary images https://review.opendev.org/c/openstack/kolla/+/820371 | 16:21 |
| admiyo | The issue with nova seems to be that the libvirt module is missing. The complute log shows: ERROR nova.virt.driver ModuleNotFoundError: No module named 'libvirt' and If I docker run bash on the container image then start python, import libvirt fails. | 16:47 |
| admiyo | sounds like an issue in building the containers? | 16:47 |
| hrw | ah. you used my patch? it was not tested and broken | 16:50 |
| admiyo | IS that the cause of the missing package? | 16:52 |
| hrw | yeah | 16:52 |
| admiyo | I did apply your patch, which sounds like it was "necessary but not sufficient" | 16:52 |
| hrw | sorry for that. idea was good but needs more work | 16:52 |
| admiyo | if I hack the Dockerfile fopr nova copute to put python3-libvirt in there, will that provide more info? | 16:53 |
| hrw | revert https://review.opendev.org/c/openstack/kolla/+/826906 in your tree | 16:53 |
| admiyo | Will do | 16:53 |
| opendevreview | Verification of a change to openstack/kolla stable/victoria failed: Drop CentOS Linux 8 image builds https://review.opendev.org/c/openstack/kolla/+/827608 | 16:54 |
| admiyo | And then what? I'll fall back to the old problem | 16:54 |
| admiyo | ah...but I se there is a change in wallaby | 16:54 |
| admiyo | hrw, SO just revert and rebase to origin/stable/wallaby, or is there still an issue? | 16:57 |
| opendevreview | Marcin Juszkiewicz proposed openstack/kolla master: nova-compute: 'ndctl' is available on all archs https://review.opendev.org/c/openstack/kolla/+/827885 | 16:58 |
| hrw | admiyo: it was pynacl failing to build for debian in wallaby - do I remember correctly? | 16:59 |
| admiyo | um...i seem to be suffereing from mid term memory loss | 17:00 |
| admiyo | yes | 17:00 |
| admiyo | and I totally did not read up to our last conversation to refresh my memory | 17:00 |
| hrw | started debian/source wallaby/aarch64 build locally to check | 17:04 |
| hrw | but weekend starts, FOSDEM during and next week I am off | 17:05 |
| edebeste | Hi all, not sure if this is more a broader OpenStack related issue, but I have an issue with Cinder and LVM. The deployment is stock Wallaby, containers are built using Kolla. Compute nodes have a partition with a cinder-volumes VG on it. The deployment is successful with no issues. Can create volumes, but when I try to move volumes from one host | 17:08 |
| edebeste | to another, I get the following: https://paste.openstack.org/show/812527/. This *only* happens when a new volume is created and NOT USED yet. The moment it's been attached to a VM and detatched, then migration works.. But attaching and detatching leads to `multipathd is not running: exit code None: | 17:08 |
| edebeste | oslo_concurrency.processutils.ProcessExecutionError: [Errno 2] No such file or directory` Is this expected behaviour? | 17:08 |
| spatel | folks, i have config override question and look like i am doing something wrong so please some one correct me here - https://paste.opendev.org/show/812528/ | 17:11 |
| spatel | Try to push out small change to specific compute node using IP like this but it doesn't work - /etc/kolla/config/nova/192.168.75.199/nova.conf | 17:12 |
| spatel | I tried compute hostname also but that doesn't help /etc/kolla/config/nova/COMP01.local/nova.conf | 17:12 |
| admiyo | hrw is this problem specific to Wallaby? If I move to Xena will I have a path forward? | 17:33 |
| opendevreview | Mark Goddard proposed openstack/kayobe master: DNM: enable iPXE for UEFI https://review.opendev.org/c/openstack/kayobe/+/827667 | 17:46 |
| opendevreview | Maksim Malchuk proposed openstack/kayobe master: Multiple DHCP Subnets for Ironic Inspector https://review.opendev.org/c/openstack/kayobe/+/827653 | 17:47 |
| ozzzo_work | I enabled durable queues on a lab cluster, and it's throwing the error: inequivalent arg 'durable' for exchange 'openstack' in vhost '/': received 'true' but current is 'false' | 17:55 |
| ozzzo_work | in my other lab cluster I fixed this by deleting the RMQ containers and volumes on all 3 controllers, but that doesn't seem to work here. How can I update the "durable" value? | 17:56 |
| ozzzo_work | I'm seeing these RMQ errors in my cinder_volume log | 17:57 |
| hrw | admiyo: you may switch to ubuntu or centos as a base in wallaby | 18:14 |
| hrw | admiyo: and in meantime please open a bug on https://bugs.launchpad.net/kolla/ about it | 18:15 |
| * hrw off | 18:15 | |
| ozzzo_work | do I need to delete my cinder containers to make them pick up the new queues? | 18:44 |
| ozzzo_work | I'm not 100% clear on where the data is stored. Can I delete containers and container volumes without losing data? | 18:45 |
| opendevreview | Merged openstack/kayobe master: Drop kolla-host.yml playbook https://review.opendev.org/c/openstack/kayobe/+/825143 | 18:51 |
| opendevreview | Radosław Piliszek proposed openstack/kolla master: collectd: pcie-errors is x86-64 only now https://review.opendev.org/c/openstack/kolla/+/827883 | 20:39 |
| spatel | Any idea why my override isn't working ? | 21:16 |
| opendevreview | Merged openstack/kolla stable/victoria: Drop CentOS Linux 8 image builds https://review.opendev.org/c/openstack/kolla/+/827608 | 21:31 |
| opendevreview | Radosław Piliszek proposed openstack/kolla-ansible master: [CI] Replace parted with lsblk https://review.opendev.org/c/openstack/kolla-ansible/+/827928 | 21:40 |
| opendevreview | Radosław Piliszek proposed openstack/kolla-ansible master: [DNM] Test kolla-ansible-ubuntu-source-zun job https://review.opendev.org/c/openstack/kolla-ansible/+/827512 | 21:44 |
| opendevreview | Radosław Piliszek proposed openstack/kolla-ansible master: [CI] Add kolla-ansible-debian-source-zun job https://review.opendev.org/c/openstack/kolla-ansible/+/827536 | 21:44 |
| opendevreview | Radosław Piliszek proposed openstack/kolla-ansible master: [CI] Test Glance interaction with Cinder https://review.opendev.org/c/openstack/kolla-ansible/+/827264 | 21:45 |
| opendevreview | James Kirsch proposed openstack/kolla master: Add LetsEncrypt images for cert request/renewal https://review.opendev.org/c/openstack/kolla/+/741339 | 22:55 |
| opendevreview | Merged openstack/kolla master: collectd: pcie-errors is x86-64 only now https://review.opendev.org/c/openstack/kolla/+/827883 | 23:29 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!