| opendevreview | Pierre Riteau proposed openstack/kolla master: Remove ipxe code for Ubuntu Focal https://review.opendev.org/c/openstack/kolla/+/873875 | 08:10 |
|---|---|---|
| mgoddard | Looks like zun jobs are failing after docker 23.0.0 | 09:39 |
| mgoddard | https://a72bab7d5d7b58439e32-2ea4ab35fcc993920ff2033895dde0a5.ssl.cf5.rackcdn.com/852392/6/check/kolla-ansible-debian-zun/3734c0a/primary/logs/system_logs/docker.txt | 09:39 |
| mgoddard | zun uses the now removed 'cluster-store' option in daemon.json | 09:39 |
| mnasiadka | yes, that's known | 09:41 |
| mnasiadka | raised a bug in zun | 09:41 |
| mnasiadka | let me pin docker in those jobs | 09:41 |
| mgoddard | https://bugs.launchpad.net/zun/+bug/2007142 | 09:41 |
| mgoddard | I see you raised it already | 09:41 |
| mnasiadka | I don't think it's going to get fixed soon ;-) | 09:42 |
| mgoddard | drop zun jobs for now? | 09:43 |
| mnasiadka | isn't it better to pin those jobs to docker 22.*? | 09:46 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: pin zun jobs to Docker 22 https://review.opendev.org/c/openstack/kolla-ansible/+/873888 | 09:51 |
| priteau | Hello. Has anyone seen this horizon build issue before? https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_046/873875/1/check/kolla-build-rocky9/0467a01/kolla/build/000_FAILED_horizon.log | 09:52 |
| mnasiadka | yes, me on RL9 yoga backport | 09:56 |
| mnasiadka | pip 23 is out, maybe that's the cause | 09:57 |
| mnasiadka | let me try pinning on a local env | 10:00 |
| mnasiadka | nope, that's not pip | 10:12 |
| mnasiadka | it's bloody setuptools | 10:18 |
| mnasiadka | let me pin | 10:18 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: Pin setuptools=67.2.* https://review.opendev.org/c/openstack/kolla/+/873894 | 10:25 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: pin zun jobs to Docker 22 https://review.opendev.org/c/openstack/kolla-ansible/+/873888 | 10:33 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: pin zun jobs to Docker 20 https://review.opendev.org/c/openstack/kolla-ansible/+/873888 | 10:52 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: pin zun jobs to Docker 20 https://review.opendev.org/c/openstack/kolla-ansible/+/873888 | 10:53 |
| hrw | mnasiadka: can you share link to failed horizon? | 11:05 |
| hrw | ah, sorry | 11:05 |
| mnasiadka | yeah, I raised an issue in setuptools github | 11:05 |
| mnasiadka | it's linked on the whiteboard | 11:05 |
| hrw | mnasiadka: thx | 11:07 |
| hrw | mnasiadka: can you check does it happen same way if 'wheel' is also installed? | 11:08 |
| mnasiadka | hrw: checked locally, it does | 11:10 |
| hrw | xstatic being 3y old can also be a problem | 11:10 |
| hrw | mnasiadka: passed with 'pip install -U wheel': https://pastebin.com/MvZ3zjbs | 11:12 |
| hrw | (venv) [root@38baaa1a52e3 ~]# pip list |grep horizon | 11:13 |
| hrw | horizon 23.1.0.dev66 | 11:13 |
| hrw | meh, horizon... is it maintained at all? | 11:14 |
| mnasiadka | hrw: xstatic repo on opendev has newer versions, but it seems those are not uploaded to pypi ;-) | 11:20 |
| mnasiadka | hrw: but we're updating wheel in openstack-base, so why the heck it's not working? :) | 11:22 |
| hrw | let me loook | 11:24 |
| hrw | no idea | 11:30 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: pin zun jobs to Docker 20 https://review.opendev.org/c/openstack/kolla-ansible/+/873888 | 11:31 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: cephadm: use docker by default https://review.opendev.org/c/openstack/kolla-ansible/+/870093 | 11:47 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: cephadm: use docker by default https://review.opendev.org/c/openstack/kolla-ansible/+/870093 | 11:51 |
| mnasiadka | hrw: I guess let's pin for now, unless you have a better idea | 11:58 |
| mnasiadka | hrw, bbezak, mgoddard: https://review.opendev.org/c/openstack/kolla/+/873894 - zuul passed green | 12:02 |
| kevko | hh, i was the person who uploaded xstatic-moment to opendev i think :D | 12:04 |
| kevko | and pypi also | 12:05 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: cephadm: use docker by default https://review.opendev.org/c/openstack/kolla-ansible/+/870093 | 12:08 |
| guesswhat[m] | Is https://github.com/yoctozepto usually in this channel ? | 12:20 |
| kevko | guesswhat[m]: used to be ..but now probably not anymore :P | 12:31 |
| guesswhat[m] | @kevko oh, thats a pitty, seems he know how to integrate Cinder and Glance https://review.opendev.org/c/openstack/kolla-ansible/+/699910 | 12:49 |
| kevko | guesswhat[m]: do you have enable_cinder_backend_iscsi enabled ? | 12:51 |
| guesswhat[m] | yes, nova->cinder->iscsi->lvm is working correctly | 12:52 |
| guesswhat[m] | got some troubles with connecting to iscsi portal from glance, see https://bugs.launchpad.net/kolla/+bug/1869072 | 12:53 |
| guesswhat[m] | unfortunately there is nobody who is willing to help me | 12:53 |
| guesswhat[m] | ehm, wrong bug report, this one is correct https://bugs.launchpad.net/kolla-ansible/+bug/2007261 | 12:56 |
| kevko | guesswhat[m]: i will take a look on it ...but now i don't have a time ..maybe next week :( | 12:57 |
| kevko | i've already seen this bugreport | 12:57 |
| guesswhat[m] | iscsiadm error is not helpful, even error code 20 | 12:58 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: cephadm: use docker by default https://review.opendev.org/c/openstack/kolla-ansible/+/870093 | 13:43 |
| mnasiadka | mgoddard mnasiadka hrw bbezak frickler kevko - meeting in 10 minutes | 13:50 |
| opendevreview | Merged openstack/kolla master: Pin setuptools=67.2.* https://review.opendev.org/c/openstack/kolla/+/873894 | 13:58 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/zed: Pin setuptools=67.2.* https://review.opendev.org/c/openstack/kolla/+/873912 | 13:59 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/yoga: Pin setuptools=67.2.* https://review.opendev.org/c/openstack/kolla/+/873913 | 13:59 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/xena: Pin setuptools=67.2.* https://review.opendev.org/c/openstack/kolla/+/873914 | 13:59 |
| opendevreview | Pierre Riteau proposed openstack/kolla master: Remove ipxe code for Ubuntu Focal https://review.opendev.org/c/openstack/kolla/+/873875 | 14:00 |
| kevko | \o/ | 14:00 |
| mmalchuk | \o | 14:00 |
| mnasiadka | #startmeeting kolla | 14:00 |
| opendevmeet | Meeting started Wed Feb 15 14:00:59 2023 UTC and is due to finish in 60 minutes. The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot. | 14:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 14:00 |
| opendevmeet | The meeting name has been set to 'kolla' | 14:00 |
| mnasiadka | kevko: mmalchuk - now ;) | 14:01 |
| mnasiadka | o/ | 14:01 |
| mnasiadka | oops | 14:01 |
| mnasiadka | #topic rollcall | 14:01 |
| mnasiadka | now | 14:01 |
| mnasiadka | o/ | 14:01 |
| kevko | \o/ | 14:01 |
| mhiner | o/ | 14:01 |
| bbezak | o/ | 14:01 |
| frickler | o( | 14:02 |
| frickler | oops, off-by-one ;) | 14:02 |
| kevko | #help | 14:03 |
| mnasiadka | #topic agenda | 14:03 |
| mnasiadka | * Review action items from the last meeting | 14:03 |
| mnasiadka | * CI status | 14:03 |
| mnasiadka | * Release tasks | 14:03 |
| mnasiadka | * Regular stable releases (first meeting in a month) | 14:03 |
| mnasiadka | * Current cycle planning | 14:03 |
| mnasiadka | * Additional agenda (from whiteboard) | 14:03 |
| mnasiadka | * Open discussion | 14:03 |
| mnasiadka | #topic Review action items from the last meeting | 14:03 |
| ramona-beermann[m] | o/ | 14:03 |
| mnasiadka | bbezak was to upload monthly releases | 14:03 |
| mnasiadka | done | 14:03 |
| bbezak | I did | 14:03 |
| mnasiadka | #topic CI status | 14:03 |
| mnasiadka | zun CI broken by Docker 23 - pursuing a pin to 20.* | 14:03 |
| mnasiadka | setuptools broke Horizon builds - pinning to previous release (and backporting) | 14:03 |
| mnasiadka | #topic Release tasks | 14:04 |
| mnasiadka | cycle highlights behind the corner | 14:04 |
| mnasiadka | (as in next week) | 14:04 |
| mnasiadka | nothing else on the release tasks calendar for now | 14:04 |
| mnasiadka | #topic Current cycle planning | 14:05 |
| mnasiadka | Ok then, let's have a look on the priorities | 14:05 |
| mnasiadka | mhiner: how is podman? | 14:05 |
| mhiner | main patch is still in merge conflict but ContainerWorker refactor is ready | 14:06 |
| mnasiadka | there's also let's encrypt that would be nice to land - but it's in merge conflict (https://review.opendev.org/c/openstack/kolla-ansible/+/741340) | 14:06 |
| mnasiadka | if there's anybody willing to rebase and pick it up - would be nice | 14:06 |
| kevko | mnasiadka: i will take an eye on it | 14:07 |
| mnasiadka | nothing else on the prio list | 14:07 |
| mnasiadka | no additional agenda on the whiteboard | 14:07 |
| mnasiadka | #topic open discussion | 14:08 |
| mnasiadka | if anybody has patches that he would like to prioritise reviewing, or any other discussion - now is the time :) | 14:08 |
| mmalchuk | https://review.opendev.org/c/openstack/kolla/+/872005 | 14:08 |
| mmalchuk | mnasiadka have a look? | 14:09 |
| mnasiadka | yeah, I remember that | 14:09 |
| mnasiadka | no other option I guess? | 14:09 |
| mmalchuk | ok | 14:10 |
| mnasiadka | hrw: https://review.opendev.org/c/openstack/kolla/+/872005 - time to merge I guess | 14:10 |
| mmalchuk | this is not kolla/kayobe, but would be great to review: | 14:13 |
| mmalchuk | https://review.opendev.org/c/openstack/diskimage-builder/+/872430 | 14:13 |
| mmalchuk | https://review.opendev.org/c/openstack/diskimage-builder/+/869270 | 14:13 |
| mmalchuk | thx a lot | 14:13 |
| mnasiadka | https://review.opendev.org/q/topic:assert - there's a series of patches by ebbex that we could merge (replacing fail: with assert:) | 14:14 |
| hrw | elo | 14:15 |
| hrw | https://review.opendev.org/c/openstack/kolla/+/872005 - I have a mixed feelings about this one so will hold | 14:15 |
| mnasiadka | well, I think we broke people, that build in a semi-offline environment | 14:16 |
| mnasiadka | so either we revert, or provide similar functionality | 14:16 |
| mnasiadka | the only alternative I can think of is moving .repo files to a template and adding a flag to not use public gpg keys? | 14:17 |
| hrw | if they replace *.repo files then they can add gpgkey line there. if they rely on keys being present in base/Dockerfile.j2 then template overrides exist | 14:17 |
| mnasiadka | ok, so how do they replace *.repo files without editing code? | 14:18 |
| mnasiadka | ah, rpm_setup_config | 14:18 |
| mmalchuk | we build weithout any problems before. this is not feature but fix which brings nothing new | 14:18 |
| hrw | mmalchuk: this is revert of removal of not needed code | 14:19 |
| mnasiadka | mmalchuk: have you tried providing your own .repo files? | 14:19 |
| mmalchuk | but with it we lost an ability to override repos. because of static files | 14:19 |
| hrw | mmalchuk: how do you override repos? | 14:20 |
| mmalchuk | thorugh blocks feature ofcourse | 14:20 |
| mnasiadka | https://docs.openstack.org/kolla/latest/admin/image-building.html#custom-repos - there's this | 14:21 |
| hrw | you mean template overrides? | 14:21 |
| mmalchuk | commit message provide the links to docs | 14:21 |
| mmalchuk | sure | 14:21 |
| mmalchuk | with this change the only way to exec sed in the overrides | 14:22 |
| hrw | mmalchuk: I never used kayobe | 14:22 |
| mmalchuk | sad | 14:22 |
| mmalchuk | it simplifies life) | 14:22 |
| mmalchuk | even without kayobe you can use overrides wia kollabuild parameters | 14:23 |
| hrw | mmalchuk: define block for base to override base_yum_repo_keys with own values? | 14:23 |
| mmalchuk | with this change we lost them for redhat based systems, only debian based works | 14:23 |
| mmalchuk | yep | 14:23 |
| mnasiadka | apparently kayobe only supports templates-overrides, not rpm_setup_config | 14:24 |
| hrw | mmalchuk: can you share some logs from failed run? | 14:24 |
| mnasiadka | well, at least out of the box | 14:24 |
| mmalchuk | right now can't | 14:24 |
| hrw | :( | 14:25 |
| mmalchuk | we move to ubuntu | 14:25 |
| hrw | from what I understand kayobe run is not easy to replicate when all you have is one machine, right? | 14:26 |
| mmalchuk | no | 14:26 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Add support for LetsEncrypt-managed certs https://review.opendev.org/c/openstack/kolla-ansible/+/741340 | 14:26 |
| mnasiadka | so - basically - let's revert it now, and add rpm_setup_config support to Kayobe? | 14:28 |
| mnasiadka | and then we can revert the revert? ;) | 14:28 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Add support for LetsEncrypt-managed certs https://review.opendev.org/c/openstack/kolla-ansible/+/741340 | 14:28 |
| mnasiadka | (plus add proper docs how it should be overridden) | 14:29 |
| mnasiadka | or option b - just revert and leave it as is forever | 14:29 |
| mmalchuk | mnasiadka good idea for rpm_setup_config | 14:30 |
| mnasiadka | sed is a workaround that we all have been using for long time (including Kolla CI) | 14:31 |
| mmalchuk | but I can't create it because of ubuntu | 14:31 |
| mnasiadka | well, Ubuntu/Debian is purely repos.yaml | 14:31 |
| hrw | mnasiadka: and keys fetched in base | 14:31 |
| mmalchuk | sed is a crutch ;) | 14:32 |
| mnasiadka | so maybe we should rather aim to unify that in repos.yaml? | 14:32 |
| mmalchuk | about repos.yml | 14:32 |
| mnasiadka | like write out RH style .repo (like we do with Debian/Ubuntu) | 14:32 |
| mnasiadka | and add functionality in there to download the keys? | 14:32 |
| mmalchuk | https://review.opendev.org/c/openstack/kayobe/+/845338 | 14:32 |
| mmalchuk | please have a look | 14:32 |
| mnasiadka | added review priority +1, will have a look later | 14:33 |
| mmalchuk | good idea but it take more time | 14:33 |
| mmalchuk | +1 for unify solutions based on repos.yml | 14:33 |
| mnasiadka | takes more time, makes more sense | 14:34 |
| hrw | mnasiadka: good idea | 14:34 |
| hrw | mnasiadka: and move keys from base/Dockerfile.j2 to repos.yaml | 14:34 |
| mmalchuk | sure | 14:34 |
| mnasiadka | added that to the whiteboard into priorities | 14:35 |
| mnasiadka | hrw: are you willing to look into that, or should we look for a different volunteer? | 14:35 |
| mmalchuk | I can, if we don't need this so quick) | 14:36 |
| hrw | mnasiadka: I am going for some time off so if someone can look then it would be better | 14:36 |
| mmalchuk | so assing me) | 14:36 |
| mnasiadka | I might have some time to look into that in the coming weeks | 14:37 |
| mmalchuk | assign* | 14:37 |
| mnasiadka | Anyway, I'll assign mmalchuk | 14:37 |
| mnasiadka | So - do we merge this revert? | 14:37 |
| hrw | remember that in EL systems there are several repo files which comes from packages | 14:37 |
| mmalchuk | why not | 14:38 |
| mnasiadka | yup, we only add those, that do not come from centos-extras or are not downloadable iirc | 14:38 |
| hrw | so we just enable/disable them instead of adding new repo files like we do on Debuntu world | 14:38 |
| mmalchuk | it don't break anything | 14:38 |
| mnasiadka | ok, so we have consensus | 14:38 |
| hrw | and we disable most of EL distro ones after we install their distro-release-* packages | 14:38 |
| hrw | for example CRB or EPEL are installed from distro instead of being added by us (as a file) | 14:39 |
| mnasiadka | yes, special treatment needed | 14:40 |
| mnasiadka | but it's doable | 14:40 |
| hrw | and it also solves dependencies for us - centos-release-openstack-yoga brings few other centos-release-* packages so we do not have to list them to install (but list to disable) | 14:40 |
| mnasiadka | ok, unfortunately I have to run | 14:41 |
| mnasiadka | feel free to continue the discussion | 14:41 |
| mnasiadka | #endmeeting | 14:41 |
| opendevmeet | Meeting ended Wed Feb 15 14:41:31 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 14:41 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/kolla/2023/kolla.2023-02-15-14.00.html | 14:41 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/kolla/2023/kolla.2023-02-15-14.00.txt | 14:41 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/kolla/2023/kolla.2023-02-15-14.00.log.html | 14:41 |
| mmalchuk | so many dependencies and cases) | 14:41 |
| hrw | and there is this moment when stable gets released, we stop using delorean repos and their weird renames of centos repos | 14:41 |
| mnasiadka | Thanks for coming! | 14:41 |
| mmalchuk | mnasiadka thx | 14:41 |
| mmalchuk | we need to replace all crutches like sed to the variables and blocks | 14:43 |
| mnasiadka | would be nice | 14:43 |
| opendevreview | Merged openstack/kolla-ansible master: docs: fix information about libvirt SASL auth https://review.opendev.org/c/openstack/kolla-ansible/+/872636 | 14:46 |
| opendevreview | Pierre Riteau proposed openstack/kolla-ansible stable/zed: docs: fix information about libvirt SASL auth https://review.opendev.org/c/openstack/kolla-ansible/+/873915 | 15:41 |
| opendevreview | Pierre Riteau proposed openstack/kolla-ansible stable/yoga: docs: fix information about libvirt SASL auth https://review.opendev.org/c/openstack/kolla-ansible/+/873916 | 15:42 |
| opendevreview | Pierre Riteau proposed openstack/kolla-ansible stable/xena: docs: fix information about libvirt SASL auth https://review.opendev.org/c/openstack/kolla-ansible/+/873917 | 15:42 |
| opendevreview | Pierre Riteau proposed openstack/kolla-ansible stable/wallaby: docs: fix information about libvirt SASL auth https://review.opendev.org/c/openstack/kolla-ansible/+/873918 | 15:42 |
| hrw | mnasiadka: I upgraded aarch64 EL8/9 builds of erlang to 25.2 | 15:46 |
| hrw | to be in sync | 15:46 |
| mnasiadka | hrw: goodie | 16:03 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/yoga: Add RockyLinux 9 support https://review.opendev.org/c/openstack/kolla/+/873549 | 16:08 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: cephadm: use docker by default https://review.opendev.org/c/openstack/kolla-ansible/+/870093 | 16:30 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/yoga: Add RockyLinux 9 support https://review.opendev.org/c/openstack/kolla/+/873549 | 16:45 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Add support for LetsEncrypt-managed certs https://review.opendev.org/c/openstack/kolla-ansible/+/741340 | 16:52 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: cephadm: use docker by default https://review.opendev.org/c/openstack/kolla-ansible/+/870093 | 16:56 |
| spatel | kevko any idea what is going on here - https://paste.opendev.org/show/bMCHGuGzRocZ8jIMwGGB/ | 16:59 |
| spatel | everything working fine but just getting this logs | 16:59 |
| pvh_sa__ | hello kolla folks - I have a rather ancient install (queens) - which has been the victim of repeated power failures in recent months. perhaps due to that some of the libvirt XML files seem to have gone missing. the servers are listed in `openstack server list` but the libvirt XML files mentioned in `openstack server show` are missing (I checked /var/lib/docker/volumes/nova_libvirt_qemu/_data) | 17:05 |
| pvh_sa__ | is there a way to programmatically recreate these missing files? | 17:05 |
| pvh_sa__ | (and yes I know queens is long past end-of-life - upgrading this install is on my TODO list but I need to get it in a working state first) | 17:07 |
| spatel | I am having this error as mentioned in bug after my ceph nodes rebooted :( - https://bugs.launchpad.net/kolla/+bug/1688496 | 18:01 |
| spatel | I am running ubuntu 22.04 | 18:01 |
| opendevreview | Dawud proposed openstack/kolla master: Add comprehensive checks for container restarts https://review.opendev.org/c/openstack/kolla/+/873972 | 18:02 |
| supamatt | is there a good way to use ovs bonds with kolla? | 18:58 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!