Wednesday, 2023-02-15

opendevreviewPierre Riteau proposed openstack/kolla master: Remove ipxe code for Ubuntu Focal  https://review.opendev.org/c/openstack/kolla/+/87387508:10
mgoddardLooks like zun jobs are failing after docker 23.0.009:39
mgoddardhttps://a72bab7d5d7b58439e32-2ea4ab35fcc993920ff2033895dde0a5.ssl.cf5.rackcdn.com/852392/6/check/kolla-ansible-debian-zun/3734c0a/primary/logs/system_logs/docker.txt09:39
mgoddardzun uses the now removed 'cluster-store' option in daemon.json09:39
mnasiadkayes, that's known09:41
mnasiadkaraised a bug in zun09:41
mnasiadkalet me pin docker in those jobs09:41
mgoddardhttps://bugs.launchpad.net/zun/+bug/200714209:41
mgoddardI see you raised it already09:41
mnasiadkaI don't think it's going to get fixed soon ;-)09:42
mgoddarddrop zun jobs for now?09:43
mnasiadkaisn't it better to pin those jobs to docker 22.*?09:46
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: CI: pin zun jobs to Docker 22  https://review.opendev.org/c/openstack/kolla-ansible/+/87388809:51
priteauHello. Has anyone seen this horizon build issue before? https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_046/873875/1/check/kolla-build-rocky9/0467a01/kolla/build/000_FAILED_horizon.log09:52
mnasiadkayes, me on RL9 yoga backport09:56
mnasiadkapip 23 is out, maybe that's the cause09:57
mnasiadkalet me try pinning on a local env10:00
mnasiadkanope, that's not pip10:12
mnasiadkait's bloody setuptools10:18
mnasiadkalet me pin10:18
opendevreviewMichal Nasiadka proposed openstack/kolla master: Pin setuptools=67.2.*  https://review.opendev.org/c/openstack/kolla/+/87389410:25
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: CI: pin zun jobs to Docker 22  https://review.opendev.org/c/openstack/kolla-ansible/+/87388810:33
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: CI: pin zun jobs to Docker 20  https://review.opendev.org/c/openstack/kolla-ansible/+/87388810:52
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: CI: pin zun jobs to Docker 20  https://review.opendev.org/c/openstack/kolla-ansible/+/87388810:53
hrwmnasiadka: can you share link to failed horizon?11:05
hrwah, sorry11:05
mnasiadkayeah, I raised an issue in setuptools github11:05
mnasiadkait's linked on the whiteboard11:05
hrwmnasiadka: thx11:07
hrwmnasiadka: can you check does it happen same way if 'wheel' is also installed?11:08
mnasiadkahrw: checked locally, it does11:10
hrwxstatic being 3y old can also be a problem11:10
hrwmnasiadka: passed with 'pip install -U wheel': https://pastebin.com/MvZ3zjbs11:12
hrw(venv) [root@38baaa1a52e3 ~]# pip list |grep horizon11:13
hrwhorizon                      23.1.0.dev6611:13
hrwmeh, horizon... is it maintained at all?11:14
mnasiadkahrw: xstatic repo on opendev has newer versions, but it seems those are not uploaded to pypi ;-)11:20
mnasiadkahrw: but we're updating wheel in openstack-base, so why the heck it's not working? :)11:22
hrwlet me loook11:24
hrwno idea11:30
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: CI: pin zun jobs to Docker 20  https://review.opendev.org/c/openstack/kolla-ansible/+/87388811:31
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: CI: cephadm: use docker by default  https://review.opendev.org/c/openstack/kolla-ansible/+/87009311:47
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: CI: cephadm: use docker by default  https://review.opendev.org/c/openstack/kolla-ansible/+/87009311:51
mnasiadkahrw: I guess let's pin for now, unless you have a better idea11:58
mnasiadkahrw, bbezak, mgoddard: https://review.opendev.org/c/openstack/kolla/+/873894 - zuul passed green12:02
kevkohh, i was the person who uploaded xstatic-moment to opendev i think :D 12:04
kevkoand pypi also12:05
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: CI: cephadm: use docker by default  https://review.opendev.org/c/openstack/kolla-ansible/+/87009312:08
guesswhat[m]Is https://github.com/yoctozepto usually in this channel  ?12:20
kevkoguesswhat[m]: used to be ..but now probably not anymore :P 12:31
guesswhat[m]@kevko oh, thats a pitty, seems he know how to integrate Cinder and Glance https://review.opendev.org/c/openstack/kolla-ansible/+/69991012:49
kevkoguesswhat[m]: do you have enable_cinder_backend_iscsi enabled  ? 12:51
guesswhat[m]yes, nova->cinder->iscsi->lvm is working correctly12:52
guesswhat[m]got some troubles with connecting to iscsi portal from glance, see https://bugs.launchpad.net/kolla/+bug/186907212:53
guesswhat[m]unfortunately there is nobody who is willing to help me12:53
guesswhat[m]ehm, wrong bug report, this one is correct https://bugs.launchpad.net/kolla-ansible/+bug/200726112:56
kevkoguesswhat[m]: i will take a look on it ...but now i don't have a time ..maybe next week :( 12:57
kevkoi've already seen this bugreport 12:57
guesswhat[m]iscsiadm error is not helpful, even error code 2012:58
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: CI: cephadm: use docker by default  https://review.opendev.org/c/openstack/kolla-ansible/+/87009313:43
mnasiadkamgoddard mnasiadka hrw  bbezak frickler kevko - meeting in 10 minutes13:50
opendevreviewMerged openstack/kolla master: Pin setuptools=67.2.*  https://review.opendev.org/c/openstack/kolla/+/87389413:58
opendevreviewMichal Nasiadka proposed openstack/kolla stable/zed: Pin setuptools=67.2.*  https://review.opendev.org/c/openstack/kolla/+/87391213:59
opendevreviewMichal Nasiadka proposed openstack/kolla stable/yoga: Pin setuptools=67.2.*  https://review.opendev.org/c/openstack/kolla/+/87391313:59
opendevreviewMichal Nasiadka proposed openstack/kolla stable/xena: Pin setuptools=67.2.*  https://review.opendev.org/c/openstack/kolla/+/87391413:59
opendevreviewPierre Riteau proposed openstack/kolla master: Remove ipxe code for Ubuntu Focal  https://review.opendev.org/c/openstack/kolla/+/87387514:00
kevko\o/14:00
mmalchuk\o14:00
mnasiadka#startmeeting kolla14:00
opendevmeetMeeting started Wed Feb 15 14:00:59 2023 UTC and is due to finish in 60 minutes.  The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot.14:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.14:00
opendevmeetThe meeting name has been set to 'kolla'14:00
mnasiadkakevko: mmalchuk - now ;)14:01
mnasiadkao/14:01
mnasiadkaoops14:01
mnasiadka#topic rollcall14:01
mnasiadkanow14:01
mnasiadkao/14:01
kevko\o/ 14:01
mhinero/14:01
bbezako/14:01
fricklero(14:02
frickleroops, off-by-one ;)14:02
kevko#help14:03
mnasiadka#topic agenda14:03
mnasiadka* Review action items from the last meeting14:03
mnasiadka* CI status14:03
mnasiadka* Release tasks14:03
mnasiadka* Regular stable releases (first meeting in a month)14:03
mnasiadka* Current cycle planning14:03
mnasiadka* Additional agenda (from whiteboard)14:03
mnasiadka* Open discussion14:03
mnasiadka#topic Review action items from the last meeting14:03
ramona-beermann[m]o/14:03
mnasiadkabbezak was to upload monthly releases14:03
mnasiadkadone14:03
bbezakI did14:03
mnasiadka#topic CI status14:03
mnasiadkazun CI broken by Docker 23 - pursuing a pin to 20.*14:03
mnasiadkasetuptools broke Horizon builds - pinning to previous release (and backporting)14:03
mnasiadka#topic Release tasks14:04
mnasiadkacycle highlights behind the corner14:04
mnasiadka(as in next week)14:04
mnasiadkanothing else on the release tasks calendar for now14:04
mnasiadka#topic Current cycle planning14:05
mnasiadkaOk then, let's have a look on the priorities14:05
mnasiadkamhiner: how is podman?14:05
mhinermain patch is still in merge conflict but ContainerWorker refactor is ready14:06
mnasiadkathere's also let's encrypt that would be nice to land - but it's in merge conflict (https://review.opendev.org/c/openstack/kolla-ansible/+/741340)14:06
mnasiadkaif there's anybody willing to rebase and pick it up - would be nice14:06
kevkomnasiadka: i will take an eye on it14:07
mnasiadkanothing else on the prio list14:07
mnasiadkano additional agenda on the whiteboard14:07
mnasiadka#topic open discussion14:08
mnasiadkaif anybody has patches that he would like to prioritise reviewing, or any other discussion - now is the time :)14:08
mmalchukhttps://review.opendev.org/c/openstack/kolla/+/87200514:08
mmalchukmnasiadka have a look?14:09
mnasiadkayeah, I remember that14:09
mnasiadkano other option I guess?14:09
mmalchukok14:10
mnasiadkahrw: https://review.opendev.org/c/openstack/kolla/+/872005 - time to merge I guess14:10
mmalchukthis is not kolla/kayobe, but would be great to review:14:13
mmalchukhttps://review.opendev.org/c/openstack/diskimage-builder/+/87243014:13
mmalchukhttps://review.opendev.org/c/openstack/diskimage-builder/+/86927014:13
mmalchukthx a lot14:13
mnasiadkahttps://review.opendev.org/q/topic:assert - there's a series of patches by ebbex that we could merge (replacing fail: with assert:)14:14
hrwelo14:15
hrwhttps://review.opendev.org/c/openstack/kolla/+/872005 - I have a mixed feelings about this one so will hold14:15
mnasiadkawell, I think we broke people, that build in a semi-offline environment14:16
mnasiadkaso either we revert, or provide similar functionality14:16
mnasiadkathe only alternative I can think of is moving .repo files to a template and adding a flag to not use public gpg keys?14:17
hrwif they replace *.repo files then they can add gpgkey line there. if they rely on keys being present in base/Dockerfile.j2 then template overrides exist14:17
mnasiadkaok, so how do they replace *.repo files without editing code? 14:18
mnasiadkaah, rpm_setup_config14:18
mmalchukwe build weithout any problems before. this is not feature but fix which brings nothing new14:18
hrwmmalchuk: this is revert of removal of not needed code14:19
mnasiadkammalchuk: have you tried providing your own .repo files?14:19
mmalchukbut with it we lost an ability to override repos. because of static files14:19
hrwmmalchuk: how do you override repos?14:20
mmalchukthorugh blocks feature ofcourse14:20
mnasiadkahttps://docs.openstack.org/kolla/latest/admin/image-building.html#custom-repos - there's this14:21
hrwyou mean template overrides?14:21
mmalchukcommit message provide the links to docs14:21
mmalchuksure14:21
mmalchukwith this change the only way to exec sed in the overrides14:22
hrwmmalchuk: I never used kayobe14:22
mmalchuksad14:22
mmalchukit simplifies life)14:22
mmalchukeven without kayobe you can use overrides wia kollabuild parameters14:23
hrwmmalchuk: define block for base to override base_yum_repo_keys with own values?14:23
mmalchukwith this change we lost them for redhat based systems, only debian based works14:23
mmalchukyep14:23
mnasiadkaapparently kayobe only supports templates-overrides, not rpm_setup_config14:24
hrwmmalchuk: can you share some logs from failed run?14:24
mnasiadkawell, at least out of the box14:24
mmalchukright now can't14:24
hrw:(14:25
mmalchukwe move to ubuntu14:25
hrwfrom what I understand kayobe run is not easy to replicate when all you have is one machine, right?14:26
mmalchukno14:26
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Add support for LetsEncrypt-managed certs  https://review.opendev.org/c/openstack/kolla-ansible/+/74134014:26
mnasiadkaso - basically - let's revert it now, and add rpm_setup_config support to Kayobe?14:28
mnasiadkaand then we can revert the revert? ;)14:28
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Add support for LetsEncrypt-managed certs  https://review.opendev.org/c/openstack/kolla-ansible/+/74134014:28
mnasiadka(plus add proper docs how it should be overridden)14:29
mnasiadkaor option b - just revert and leave it as is forever14:29
mmalchukmnasiadka good idea for rpm_setup_config14:30
mnasiadkased is a workaround that we all have been using for long time (including Kolla CI)14:31
mmalchukbut I can't create it because of ubuntu14:31
mnasiadkawell, Ubuntu/Debian is purely repos.yaml14:31
hrwmnasiadka: and keys fetched in base14:31
mmalchuksed is a crutch ;)14:32
mnasiadkaso maybe we should rather aim to unify that in repos.yaml?14:32
mmalchukabout repos.yml14:32
mnasiadkalike write out RH style .repo (like we do with Debian/Ubuntu)14:32
mnasiadkaand add functionality in there to download the keys?14:32
mmalchukhttps://review.opendev.org/c/openstack/kayobe/+/84533814:32
mmalchukplease have a look14:32
mnasiadkaadded review priority +1, will have a look later14:33
mmalchukgood idea but it take more time14:33
mmalchuk+1 for unify solutions based on repos.yml14:33
mnasiadkatakes more time, makes more sense14:34
hrwmnasiadka: good idea14:34
hrwmnasiadka: and move keys from base/Dockerfile.j2 to repos.yaml14:34
mmalchuksure14:34
mnasiadkaadded that to the whiteboard into priorities14:35
mnasiadkahrw: are you willing to look into that, or should we look for a different volunteer?14:35
mmalchukI can, if we don't need this so quick)14:36
hrwmnasiadka: I am going for some time off so if someone can look then it would be better14:36
mmalchukso assing me)14:36
mnasiadkaI might have some time to look into that in the coming weeks14:37
mmalchukassign*14:37
mnasiadkaAnyway, I'll assign mmalchuk 14:37
mnasiadkaSo - do we merge this revert?14:37
hrwremember that in EL systems there are several repo files which comes from packages14:37
mmalchukwhy not14:38
mnasiadkayup, we only add those, that do not come from centos-extras or are not downloadable iirc14:38
hrwso we just enable/disable them instead of adding new repo files like we do on Debuntu world14:38
mmalchukit don't break anything14:38
mnasiadkaok, so we have consensus14:38
hrwand we disable most of EL distro ones after we install their distro-release-* packages 14:38
hrwfor example CRB or EPEL are installed from distro instead of being added by us (as a file)14:39
mnasiadkayes, special treatment needed14:40
mnasiadkabut it's doable14:40
hrwand it also solves dependencies for us - centos-release-openstack-yoga brings few other centos-release-* packages so we do not have to list them to install (but list to disable)14:40
mnasiadkaok, unfortunately I have to run14:41
mnasiadkafeel free to continue the discussion14:41
mnasiadka#endmeeting14:41
opendevmeetMeeting ended Wed Feb 15 14:41:31 2023 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)14:41
opendevmeetMinutes:        https://meetings.opendev.org/meetings/kolla/2023/kolla.2023-02-15-14.00.html14:41
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/kolla/2023/kolla.2023-02-15-14.00.txt14:41
opendevmeetLog:            https://meetings.opendev.org/meetings/kolla/2023/kolla.2023-02-15-14.00.log.html14:41
mmalchukso many dependencies and cases)14:41
hrwand there is this moment when stable gets released, we stop using delorean repos and their weird renames of centos repos14:41
mnasiadkaThanks for coming!14:41
mmalchukmnasiadka thx14:41
mmalchukwe need to replace all crutches like sed to the variables and blocks14:43
mnasiadkawould be nice14:43
opendevreviewMerged openstack/kolla-ansible master: docs: fix information about libvirt SASL auth  https://review.opendev.org/c/openstack/kolla-ansible/+/87263614:46
opendevreviewPierre Riteau proposed openstack/kolla-ansible stable/zed: docs: fix information about libvirt SASL auth  https://review.opendev.org/c/openstack/kolla-ansible/+/87391515:41
opendevreviewPierre Riteau proposed openstack/kolla-ansible stable/yoga: docs: fix information about libvirt SASL auth  https://review.opendev.org/c/openstack/kolla-ansible/+/87391615:42
opendevreviewPierre Riteau proposed openstack/kolla-ansible stable/xena: docs: fix information about libvirt SASL auth  https://review.opendev.org/c/openstack/kolla-ansible/+/87391715:42
opendevreviewPierre Riteau proposed openstack/kolla-ansible stable/wallaby: docs: fix information about libvirt SASL auth  https://review.opendev.org/c/openstack/kolla-ansible/+/87391815:42
hrwmnasiadka: I upgraded aarch64 EL8/9 builds of erlang to 25.215:46
hrwto be in sync15:46
mnasiadkahrw: goodie16:03
opendevreviewMichal Nasiadka proposed openstack/kolla stable/yoga: Add RockyLinux 9 support  https://review.opendev.org/c/openstack/kolla/+/87354916:08
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: CI: cephadm: use docker by default  https://review.opendev.org/c/openstack/kolla-ansible/+/87009316:30
opendevreviewMichal Nasiadka proposed openstack/kolla stable/yoga: Add RockyLinux 9 support  https://review.opendev.org/c/openstack/kolla/+/87354916:45
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: Add support for LetsEncrypt-managed certs  https://review.opendev.org/c/openstack/kolla-ansible/+/74134016:52
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: CI: cephadm: use docker by default  https://review.opendev.org/c/openstack/kolla-ansible/+/87009316:56
spatelkevko any idea what is going on here - https://paste.opendev.org/show/bMCHGuGzRocZ8jIMwGGB/16:59
spateleverything working fine but just getting this logs 16:59
pvh_sa__hello kolla folks - I have a rather ancient install (queens) - which has been the victim of repeated power failures in recent months. perhaps due to that some of the libvirt XML files seem to have gone missing. the servers are listed in `openstack server list` but the libvirt XML files mentioned in `openstack server show` are missing (I checked /var/lib/docker/volumes/nova_libvirt_qemu/_data)17:05
pvh_sa__is there a way to programmatically recreate these missing files?17:05
pvh_sa__(and yes I know queens is long past end-of-life - upgrading this install is on my TODO list but I need to get it in a working state first)17:07
spatelI am having this error as mentioned in bug after my ceph nodes rebooted :( - https://bugs.launchpad.net/kolla/+bug/168849618:01
spatelI am running ubuntu 22.04 18:01
opendevreviewDawud proposed openstack/kolla master: Add comprehensive checks for container restarts  https://review.opendev.org/c/openstack/kolla/+/87397218:02
supamattis there a good way to use ovs bonds with kolla? 18:58

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!