Wednesday, 2021-08-25

ignaziocassanoHello, sorry if I ask but I am new on docker and kolla. Is there a mode to show and pull wallaby image version ? In other words....I  know every week new source images are published. If I want to show and pull the images  2 weeks old ....06:42
yoctozeptoignaziocassano: those old are gone07:00
ignaziocassanoSo, If I want to simulate an update I must wait next week ?07:02
ignaziocassanoyoctozepto: anycase I discovered that update stops vm because nova_compute lost connection with libvirtd07:03
ignaziocassanoyoctozepto: I do not know if your are reading my emails to openstack discussion list07:04
parallaxignaziocassano: you could build the images yourself using kolla-build; otherwise, you're right, tags are getting overwritten every week07:05
parallaxI always thought this is an anti-pattern in a Docker world07:06
*** rpittau|afk is now known as rpittau07:06
ignaziocassanoparallax: Ok thanks, but the problem form me is solving the issue I discussed above. 07:08
ignaziocassanoI am new in kolla.  I am testing it. At this time I am using installation made with my ansible playbooks. I never seen situation where vm go doen because libvirtd restars07:10
ignaziocassanoBut it kolla wallaby this happens every time I must update and libvirtd container needs to restart07:11
ignaziocassanoI do not know if there is a solution. I can image a workaround, updating one compute node at time e moving vm 07:13
ignaziocassanoIn emailing list someone wrote: systemd-machined previously did not work inside the contaiener so livert fell back to its generic cgroups backend, ignoring machined managing mancines without it but recently systemd-machined started working in the contianer.07:16
yoctozeptoignaziocassano: yeah, I've read those but I'm out of ideas for now :-(07:28
hrwmorning07:30
opendevreviewPiotr Parczewski proposed openstack/kolla-ansible master: Add support for configuring Elasticsearch replication  https://review.opendev.org/c/openstack/kolla-ansible/+/80595407:30
ignaziocassanoyoctozepto: I am reinstalling an then I will check the libvitd container to find if the bug is the same that Sean wrote07:30
Fl1ntmorning hrw !07:35
*** mgoddard- is now known as mgoddard07:47
hrwmgoddard: do we plan to merge apt keys change https://review.opendev.org/c/openstack/kolla/+/784923 or should it go into same bag as infra stuff?07:50
mnasiadkamorning08:06
mnasiadkahrw: we have a bag for that stuff? :)08:07
hrwmnasiadka: it is called 'abandonware'08:10
mnasiadkahrw: true, around the keys change - shouldn't we update https://docs.openstack.org/kolla/latest/admin/image-building.html#debian-ubuntu ?08:12
hrwmnasiadka: it is still valid08:14
mnasiadkahrw: but there's no mention of how to override the keys?08:15
hrwnever was08:15
hrwwe never said how to override apt keys08:16
hrwso if someone overrode them then they figured it out08:16
hrwso they will figure out again08:16
mnasiadkagreat user experience08:18
hrwapt keys need to be altered only if you add own repos with own keys. which probably no one does otherwise someone would provide some improvements there08:22
hrwIf you use copy of repos then same keys are in use. Or you follow CI way and disable checks08:22
mnasiadkaok, so we just keep that option in the dark then, until somebody comes and complains/improves that08:23
Fl1nthrw, I do08:27
Fl1ntFor now I still use template-override to do it as I didn't had time to patch it upstream.08:28
hrwFl1nt: can you share override?08:28
hrwI am curious of change08:28
Fl1ntOn my own I simply zeroed the keys arrays.08:29
Fl1ntdon't check them at all08:29
Fl1ntbecause when you're using offline mode, the only keys that are valid are the default one and the base one08:30
Fl1ntif you let say activate an extra repo08:30
hrwFl1nt: your offline repos have regenerated indexes signed with other key?08:31
Fl1ntyes08:31
Fl1ntwell, depends, we have both08:31
hrwFl1nt: and then key added to apt keys or inside of base image?08:31
Fl1ntdisabled08:32
hrwso no key checks at all? like we do on ci?08:32
Fl1ntyep08:32
Fl1ntGPG Keys management is a pain in the ass when you're offline08:32
Fl1ntlike08:32
Fl1nthere is an exemple08:32
Fl1ntfrom CentOS08:32
Fl1ntWhen you install BaseOS, the GPG Key is correct as it install on keystore08:33
Fl1nthowever08:33
hrwFl1nt: so for your use it would be good to move whole apt keys stuff to base_debuntu_apt_keys block so you could override it with empty one?08:34
Fl1ntonce you want to add new repository the "correct" way by using dnf install centos-openstack-release-x or centos-release-epel etc it won't work08:34
hrwah. it is in base_ubuntu_package_key_installation already ;d08:34
hrwFl1nt: override this block with empty08:34
mgoddardhrw: can you add it to today's meeting agenda?08:34
Fl1ntYeah, that's how I do it now08:34
hrwthen no apt keys are fetched and you do not have to empty lists08:34
hrwmgoddard: ok08:35
Fl1ntbase_<os>_key block are zeroed on template-override.08:35
hrwmgoddard: the problem is that I may not be present during meeting :(08:35
Fl1ntI'll be there today ^^08:35
Fl1ntfor once...08:36
hrwadded08:37
hrwFl1nt: centos-*-release* packages iirc provide keys inside so if you have a way of installing them offline then you have keys08:37
Fl1ntAre you sure?08:39
Fl1ntI'll recheck08:39
Fl1ntbut last time I tried it, it wasn't providing them and we had this circle loop where we install them but then need an extra step to sed the baseURL in order to get them correctly working and to correctly retrieve the RPM-GPG key.08:40
opendevreviewMark Goddard proposed openstack/kolla-ansible stable/train: DNM: test CentOS on Train  https://review.opendev.org/c/openstack/kolla-ansible/+/80597108:41
hrwFl1nt: https://koji.mbox.centos.org/koji/rpminfo?rpmID=30731508:43
hrwTASK [tensorflow : build TensorFlow 2.6.0-rc2 for cp39-cp39]08:49
hrwop08:49
hrws08:49
*** ykarel is now known as ykarel|lunch08:52
ignaziocassanoHello, anyone tried on wallaby to restart nova_libvirt container ? All vm on the nodes stop to responding and  docker exec -it nova_libvirt machinectl list reports "no machines". This is what happens also when and update is done because nova_libvirt is restarted. 08:59
hrwignaziocassano: shouldn't some kind of 'nova shutdownnode' be called first?09:01
hrwI would treat container restart same as baremetal restart09:01
hrwbut I do not manage deployments09:02
ignaziocassanohrw: I have only restart the nova libvirt container and nova compute  logs: https://paste.openstack.org/show/808301/09:04
ignaziocassanothis is because when I update the wallaby images and I run a deploy, I lost connection for all vm09:05
yoctozeptoignaziocassano: hmm, so you are specifically hit by the libvirt container restart... interesting09:08
yoctozeptoI have something you may want to try out09:08
ignaziocassanoyoctozepto: I can test....let me know09:08
yoctozeptohttps://review.opendev.org/c/openstack/kolla-ansible/+/79426209:09
yoctozeptoyou need to patch your kolla ansible with that09:09
yoctozeptothen run it and then try booting instances and restarting libvirt09:10
ignaziocassanoyoctozepto: I do not think is the bug wrote by Seean Mooney because machinectl list show running machines running it in the container and also out of container09:10
yoctozeptonote, after applying this patch, machinectl should be unable to list the instances all the time09:10
yoctozeptoignaziocassano: well, sure, but I'm just giving you a "more isolation" patch which prevents machinectl from knowing at all09:10
ignaziocassanoyoctozepto: I must apply the above patch and then run a deploy ?09:12
hrw'nova host-evacuate-live HOSTNAME', then restart/update and add it back to computes?09:12
ignaziocassano<hwr> this could be a woraround. It works but we must  try to solve this problem09:16
opendevreviewMark Goddard proposed openstack/kolla-ansible master: nova: don't fail all hosts on service registration failures  https://review.opendev.org/c/openstack/kolla-ansible/+/80559909:19
ignaziocassanoyoctozepto: I must apply the above patch and then run a deploy ?09:20
yoctozeptoignaziocassano: yes09:23
yoctozeptoafter deploy it should change the behaviour09:23
yoctozeptoyou should be able to spin up an instance AND *not* see it in machinectl09:24
yoctozeptothen try restarting libvirt09:24
yoctozeptoif you still see the machine in machinectl then the patch was not properly applied or I'm missing something else :D09:24
ignaziocassanoyoctozepto: I am running the deploy09:24
ignaziocassanoyoctozepto: after deployng with the patch hypervisors are marked down and become unresponsive. Trying to connect via ssh take a long time.09:45
yoctozeptoignaziocassano: well, the hypervisors themselves should be fine with this patch; at least that was tested ;-) 09:48
ignaziocassanoyoctozepto: libvirt.log reports: mounting "/var/lib/nova/mnt" to rootfs at "/var/lib/nova/mnt" caused: mount through procfd: no space left on device: unknown09:53
ignaziocassanobut on var I have a lot of space09:54
ignaziocassanovar and / ave a lot of free space.....any help ?10:01
ignaziocassanoyoctozepto: I am new on docker , please, why libvirtd report no space if /var have a lot of space available ?10:12
ignaziocassanopwd10:13
opendevreviewMerged openstack/kolla-ansible stable/wallaby: Remove an unused file  https://review.opendev.org/c/openstack/kolla-ansible/+/80581410:15
opendevreviewMerged openstack/kolla-ansible stable/victoria: Remove an unused file  https://review.opendev.org/c/openstack/kolla-ansible/+/80581510:15
opendevreviewMerged openstack/kolla-ansible stable/ussuri: Remove an unused file  https://review.opendev.org/c/openstack/kolla-ansible/+/80581610:15
*** ykarel|lunch is now known as ykarel10:17
amoralejhi hrw, wrt https://github.com/openstack/kolla/commit/383364e671fb82e3c0ddca37a4690a1a33be3b5410:25
hrwamoralej: yes?10:27
amoralejthis is breaking kolla in rdo :)10:33
amoralejas we use jinja package from centos10:33
amoralejand both centos stream 8 and 9 ships version 210:33
amoralejand we prefer to not duplicate packages provided in centos10:34
hrwxena openstack/requirements lists jinja 3.0.1 now10:34
amoralejyeah but, iiuc, jinja3 still can work with 2 syntax?10:35
amoraleji mean, other projects still work with >=2.x10:35
yoctozeptoignaziocassano: interesting, I have not met this issue before10:35
amoralejbut i understand are being tested with 3.0.110:35
amoralejas it's in u-c10:35
hrwamoralej: 3.0.1 does. 3.1 will not10:35
hrwamoralej: so feel free to propose revert for Xena and then we can go back to it in Yeti.10:36
amoralejso by 3.1 timeframe it will force users to update templates to 3.x specific format?10:36
hrwamoralej: no - contextfunction -> pass_context10:36
hrwamoralej: templates do not change. custom template functions need update10:37
ignaziocassanoyoctozepto: I googled for lookinf if other users are affected by this problem. I found one only with no answers. I am updating packages to verify if it solves10:37
hrwamoralej: https://jinja.palletsprojects.com/en/3.0.x/api/#jinja2.contextfunction10:37
hrwamoralej: https://review.opendev.org/c/openstack/kolla/+/798263 is what you want to revert then10:40
amoralejhrw, thanks for that info, so, it seems currently only renderspec is using it https://codesearch.opendev.org/?q=contextfunction&i=nope&literal=nope&files=&excludeFiles=&repos=10:41
amoralejhrw, you think kolla users working on rdo are using kolla rpm to build containers?10:42
amoralejor they just use the containers built by kolla proyect?10:42
hrwamoralej: no idea to be honest10:42
hrwamoralej: we usually suggest to follow git branches to build stable/* containers10:43
amoraleji was thinking if the package is being used by users...10:43
amoralejin the past was dependency for tripleo, but not now10:43
amoralejsaid this, we want to stay as a supported and friendly project for kolla10:43
hrwamoralej: do you package airship?10:44
amoralejnop10:44
amoralejbut i'm not sure if airship uses rdo10:45
amoralejwe didn't have much interaction with airship project tbh10:45
hrwok. then propose revert of jinja bump with centos repo info and I think that we can stay on jinja2 2.x for Xena10:45
hrwI did not know about airship 2 minutes ago ;D10:45
amoralej:)10:45
amoralejykarel, ^10:46
amoralejthanks hrw 10:46
ykarelThanks amoralej hrw 10:48
opendevreviewAlfredo Moralejo proposed openstack/kolla master: Revert "Update code to Jinja 3.0 style"  https://review.opendev.org/c/openstack/kolla/+/80589710:50
hrw+210:51
Fl1ntamoralej, I build kolla containers from pip with our RDO Centos based platform if it can help11:02
amoralejthanks Fl1nt, i wonder if that's the usual users pattern11:03
amoralejwhere is documented how to build containers?11:03
Fl1ntusual user pattern if you're on enterprise context11:03
Fl1ntand so you can't use kolla created images from docker hub11:03
Fl1ntamoralej, you mean this doc? https://docs.openstack.org/kolla/latest/admin/image-building.html11:04
amoralejyep11:04
amoralejso, no mention to intall kolla-build from rpm package11:05
Fl1ntjust for you to know, at least at Ubisoft, Blizzard and Govs we use the following process:11:05
Fl1ntpip install kolla == version11:05
Fl1ntthen use kolla-build from there11:05
amoralejgood to know, my question is if we would break someone by not shiping kolla package in rdo repo11:06
amoralejanyway, i'd prefer to keep it11:06
Fl1ntI can only give you feedback on my experience, but it's not representative of the whole world so I guess it's better to keep it ^^11:07
amoralejyep, i agree11:10
amoralejthanks Fl1nt anyway!11:10
Fl1ntsure you're welcome11:12
ignaziocassanoyoctozepto: the patch seems not working. I starded with a cleaned environment. an deploy does nt work : FAILED - RETRYING: Waiting for nova-compute services to register themselves (1 retries left).12:04
ignaziocassanoyoctozepto: nova_libvirt does not start and does not write anything in logs12:07
*** amoralej is now known as amoralej|lunch12:34
yoctozeptoignaziocassano: are you sure you have applied it correctly?12:42
yoctozeptobecause I know it works; I was only wondering if it fixes your issue, not if it works12:42
ignaziocassanoyoctozepto: only one question  to check if I am using the correct workflow: docker is installed on controllers and compute nodes by boostrap and/or prechecks ?12:44
yoctozeptoignaziocassano: yes, by bootstrap; you could also install it manually; the bootstrap is just a convenience ;-)12:51
ignaziocassanoI will try again to clean my anvironment and deploy again with your patch12:52
ignaziocassanoright ?12:52
yoctozeptoignaziocassano: you should not need to clean everything up though12:56
yoctozeptobut you can try12:56
yoctozeptostackhpc heads up ~> monasca leaderless https://etherpad.opendev.org/p/yoga-leaderless12:56
*** amoralej|lunch is now known as amoralej13:01
opendevreviewPiotr Parczewski proposed openstack/kolla master: Fix Elasticsearch curator  https://review.opendev.org/c/openstack/kolla/+/80599013:04
ignaziocassanoyoctozepto: libvirt container does not start : OSError: [Errno 16] Device or resource busy: '/etc/libvirt/secrets'13:13
yoctozeptoignaziocassano: you must have misapplied the patch :-(13:30
ignaziocassanoyoctozepto: I downloaded it and I copied it under /usr/local/share/kolla-ansible/ansible/roles/nova-cell/defaults 13:43
ignaziocassano yoctozepto: I moved main.yml in main.yml.orig and substitued it with the main.yml I downloaded13:45
yoctozeptoignaziocassano: ah, yeah, that will surely break13:49
yoctozeptoI will guide you then13:49
yoctozeptorestore your original file13:49
yoctozeptogo to the first line that says13:50
yoctozepto  - "/run/:/run/:shared"13:50
yoctozeptoand replace it with13:50
yoctozepto  - "/run/openvswitch:/run/openvswitch"13:50
yoctozeptogo to the second line that says13:50
yoctozepto  - "/run/:/run/:shared"13:51
yoctozeptoand remove this one13:51
yoctozeptosorry, the second line actually has less slashes:13:52
yoctozepto  - "/run:/run:shared"13:52
yoctozeptobut otherwise the instructions stay the same13:52
ignaziocassanoyoctozepto: that's all ? 13:57
yoctozeptoyes, two changes needed to be done13:59
yoctozeptoyou can now diff your original with your modified original to make sure only that has happened13:59
yoctozeptoand redo your deployment14:00
ignaziocassanoyoctozepto: let me to finish the deployment without patch.  Then I'll apply it following your istruction. It is only for be sure it is working 14:02
yoctozeptoignaziocassano: yes; libvirt should still be fine after this patch14:02
yoctozeptothis patch may help with machinectl meddling14:02
yoctozeptobut this is what we are testing :-)14:02
jd2021Hi14:21
ignaziocassanoyoctozepto: https://paste.openstack.org/show/808314/14:21
jd2021We are trying to use kolla-ansible but we have encountered some issues related with the storage backend. 14:22
jd2021Our storage is a Netapp e-series, which had an cinder driver but it has been dropped in last Openstack releases14:23
jd2021Now, it seems that we can only use LVM instead14:24
jd2021however, the Kolla documentation is not clear. It refers two approachs, the LVM backend and the LMV2/iscsi backend. 14:24
jd2021the documentation only says that to activate either one of these two backend, to ebable enable_cinder_backend_lvm: "yes"14:26
yoctozeptoignaziocassano: and also another line to remove14:28
jd2021but when doing that, it tries to use iscsi to connect to a cinder storage node, which is not the storage per se, and then connect the compute nodes with the storage node via iscsi. 14:28
ignaziocassanojd20214:28
ignaziocassanoyoctozepto: but if download the patch file and I execute the patch command is not more easy14:29
jd2021but to have the openstack storage node with a volume, it must be configured through iscsi to the netapp appliance (iscsi => lvm => iscsi)14:29
opendevreviewPiotr Parczewski proposed openstack/kolla master: Fix Elasticsearch Curator  https://review.opendev.org/c/openstack/kolla/+/80599014:30
jd2021it seems that the correct way is to just configure the same lvm volume backed by iscsi to the netapp appliance on all the compute nodes, but the kolla config does not seams to have this option available  14:30
yoctozeptoignaziocassano: yeah, that would be easier but you told me you did not use patch but replaced the files14:30
jd2021I would be very grateful if someone could shed some light onto this: How to use static block volumes via LVM directly on the compute nodes?14:32
ignaziocassanoyoctozepto: you are right. Now I download the patch file and I apply the patch. Sicne I must go out of office, I 'll contct you tomorrow14:33
yoctozeptoignaziocassano: ok, see you14:34
ignaziocassanoyoctozepto>: please chek if the patch is applyed well: https://paste.openstack.org/show/808315/14:39
*** hlm is now known as Guest537714:42
yoctozeptoignaziocassano: seems so!14:43
jd2021Can anyone?14:45
jd2021If this is not the right channel for this kind of questions, can someone point to the right one?14:45
jd2021Can anyone help?14:46
ignaziocassanoyoctozepto: see you tomorrow.  Meanwhile is anything changed for cinder. Last week the volume service was active on all controllers. Now only one the first14:49
mnasiadkajd2021: you mean that you want to expose a big disk from netapp to your storage host, and then use lvm/iscsi on the clients?14:49
mnasiadkamgoddard mnasiadka hrw egonzalez yoctozepto rafaelweingartne cosmicsound osmanlicilegi bbezak parallax Fl1nt14:50
mnasiadkameeting in 1014:50
ignaziocassanojd2021: am using netapp but with nfs protocol14:51
jd2021I would like to avoid using iscsi to connect the nova nodes to an intermediate node, which in turn connects to the netapp. 14:52
jd2021the e-series only supports iscsi, not nfs14:52
mnasiadkajd2021: you could just use iscsi to mount /var/lib/nova on the compute hosts, and use this for ephemeral storage - but for cinder I think the only option is the intermediate path14:53
jd2021the desided solution is to have a single iscsi connection from the clients (nova node) to the netapp14:53
ignaziocassanojd2021: so I think you must provide a cinder.conf on /etc/kolla/config14:53
mnasiadkajd2021: but you could confirm this with the cinder folks on #openstack-cinder14:53
jd2021I see. I'll check. But just to be clear, what is the difference between the LVM backend and LVM2/ISCSI options on kolla, since both use the same variable to be enabled?14:55
ignaziocassanokolla-ansible -i /etc/kolla/multinode -e ansible_u14:55
mnasiadkajd2021: that's the same thing14:57
jd2021ok14:57
jd2021the documentation seams to point as if they were different 14:58
ignaziocassanojd2021: I do not know If what I writing can help you: I configured iscsi for dell but out of kolla. It ceate a dell volume for each cinder volume and connect it with iscsi using multipath14:58
mnasiadkajd2021: kolla-ansible is just a deployment project, docs might be sometimes old and not accurate, but you should always refer to particular project docs - in this case cinder14:59
mnasiadka#startmeeting kolla15:00
opendevmeetMeeting started Wed Aug 25 15:00:22 2021 UTC and is due to finish in 60 minutes.  The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot.15:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:00
opendevmeetThe meeting name has been set to 'kolla'15:00
yoctozepto\o/15:00
mnasiadka#topic rollcall15:00
yoctozepto\o/15:00
mnasiadkao/15:00
mgoddard\o15:01
headphoneJameshopefully some system scope o/15:01
jd2021ignaziocassano: that's what we would like to do (dell volume example). I'll will look into it. ~15:01
parallaxo/15:01
jd2021thanks15:02
mnasiadkajd2021: we're holding a meeting now, please follow up on non-meeting topics after it ends :)15:03
mnasiadka#topic agenda15:03
yoctozeptoheadphoneJames: system scope? ;o15:03
mnasiadka* Announcements15:03
mnasiadka  ** Congratulations to mnasiadka on becoming PTL :)15:03
mnasiadka* Review action items from the last meeting15:03
mnasiadka* CI status15:03
mnasiadka* Release tasks15:03
mnasiadka* Xena cycle planning15:03
mnasiadka  ** Xena feature prioritisation https://docs.google.com/spreadsheets/d/1BuVMwP8eLnOVJDX8f3Nb6hCrNcNpRQl57T2ENU9Xao815:03
mnasiadka* handling of APT keys - https://review.opendev.org/c/openstack/kolla/+/784923 (hrw, Fl1nt)15:03
mnasiadka* Yoga PTG planning15:03
ignaziocassanojd2021: so you must look at cinder driver documentation for netapp model you are using, and the create the cinder.conf for it and put it in /etc/kolla/config/cinder.conf. I think there is also somthing to enable multipath for libvirt15:03
mnasiadka* Open discussion15:03
headphoneJamescongrads to mnasiadka!!15:03
parallax:)15:04
mnasiadka#topic Announcements15:04
yoctozeptocongrats to mnasiadka <315:04
mgoddard+115:04
mnasiadkaThanks for all the good votes, hoping mgoddard will help during the transition period :-)15:05
mgoddardof course!15:05
* yoctozepto will be there to ensure mgoddard helps15:05
yoctozepto:-)15:05
ignaziocassanojd2021: kolla add only some automation to openstack and for configuration of single cinder driver you must look at its documentation15:05
jd2021The cinder netapp driver has dropped the support for the e-series a few years ago (unified driver)15:05
yoctozeptoignaziocassano, jd2021: please don't interrupt the meeting15:05
opendevreviewMerged openstack/kolla master: Revert "Update code to Jinja 3.0 style"  https://review.opendev.org/c/openstack/kolla/+/80589715:05
ignaziocassanosorry15:05
ignaziocassanogood bye15:06
mnasiadkaOk then, let's continue15:06
mnasiadka#topic Review action items from the last meeting15:06
jd2021sorry, bye 15:06
mnasiadkaseems none outstanding from last time15:07
mnasiadkaoh, mgoddard to propose cycle highlights15:07
yoctozeptohe did not :O15:08
mgoddardhmm, I think I did not15:08
mgoddardaction me again15:08
mnasiadka#action mgoddard to propose initial cycle highlights15:08
yoctozeptomortal stack : action him!15:08
mnasiadkadid we merge the binary switch to current release? there was some problem with UCA15:09
yoctozepto(come on, no mortal kombat players?)15:09
*** amoralej is now known as amoralej|off15:10
mnasiadkayoctozepto: high school times, nearly forgot about it ;)15:10
yoctozeptomnasiadka: :D15:10
mgoddardyes, UCA is now Xena15:10
yoctozeptoI think we merged the relevant patches15:10
yoctozeptodebian will release when openstack does15:10
mnasiadkaok then15:11
mnasiadka#topic CI status15:11
mnasiadkaseems it's green-ish15:12
mnasiadkado we need to discuss any particular CI issue?15:12
mgoddardI'm not aware of any new ones15:12
yoctozeptonope15:12
yoctozeptoeverything known15:13
mnasiadkagreat to hear/see15:13
mnasiadka#topic Release tasks15:13
knockknockhello15:14
mnasiadkaR-5 cycle highlights deadline - that we have action on mgoddard 15:14
mgoddardknockknock: we're in a meeting currently, should end in ~45m15:15
knockknockI'm setting up openstack for the first time with kolla and I either misconfigured something or found a bug. It looks like a file either isn't being downloaded or is missing from the docker image. Any suggestions on figuring out if I screwed something up or if it's an actual bug?15:15
mnasiadkaR-1: Prepare for RC1 & stable branch creation - that will be next15:15
knockknockah sorry15:15
yoctozeptohow far till feature freeze15:15
yoctozeptoI might want to drop in support for consul (for masakari)15:16
mnasiadkaR-5 is feature freeze - so next week15:16
mnasiadkashould we identify features we want to push in, maybe hashtag them (?) and focus on reviewing?15:17
mgoddardcrikey, feature freeze next week15:17
mnasiadkathat's what they say here: https://releases.openstack.org/xena/schedule.html15:17
mgoddardah, that's openstack ff15:18
mgoddardwe have our own15:18
mgoddardI don't think it's included in the timeline15:19
yoctozeptomnasiadka: stop scaring poor people15:19
mgoddard"There is no official feature freeze date for the cycle-trailing model, but we aim to freeze three weeks after the common feature freeze"15:19
yoctozeptomgoddard: when do we want it then? I need ~month I guess?15:19
mgoddardR-215:19
mnasiadkamgoddard: it's not in https://docs.openstack.org/kolla/latest/contributor/release-management.html#release-schedule - should we add it there?15:19
yoctozeptothen that should be enough + asking for FFE15:19
yoctozeptoack15:19
mgoddardyes, we should add15:19
mnasiadkaany volunteer for adding? ;)15:20
mgoddardgo on then15:20
mnasiadkaok then, let me take this, should be easyfix15:22
mgoddardI meant I can do it15:22
mnasiadkaah15:22
mnasiadka#action mgoddard to add FFE on R-2 to release-management docs15:23
mnasiadkathank you mgoddard 15:23
opendevreviewMark Goddard proposed openstack/kolla master: docs: add feature freeze to release management  https://review.opendev.org/c/openstack/kolla/+/80601315:23
mgoddarddone :D15:23
mnasiadkathat's the spirit!15:23
mnasiadka#topic Xena cycle planning15:23
mnasiadkaXena feature prioritisation https://docs.google.com/spreadsheets/d/1BuVMwP8eLnOVJDX8f3Nb6hCrNcNpRQl57T2ENU9Xao815:23
mnasiadkaThis has been edited last a week ago by mgoddard, should we work into converting this into a feature list with links to Gerrit topic/changes? Or what's the plan?15:24
mgoddardmaybe this cycle we should be a bit less rigorous, with only 4 weeks until freeze?15:25
headphoneJamesAlso, in the other direction, how does one propose a feature to the list?15:25
mgoddarde.g. if you can think of a feature (or find one in that sheet) that you'd like in Xena, add it to the whiteboard15:25
mnasiadkano rigor, surely - but it would be nice for reviewers to have a ,,focus list''15:25
yoctozeptoyes, let's not use up too many sources on this part; let's try to clean up the whiteboard to get more focus15:26
yoctozeptowell, cleanup and more links and status reports15:26
yoctozeptoas delivered by kevko for proxysql15:26
mgoddardheadphoneJames: the list is all open patches as of a few weeks ago. it hasn't been massaged yet15:26
yoctozeptono massages, no fun15:27
Fl1ntsorry, get TZed15:28
Fl1nt:(15:28
mgoddardI've added some sections to the Xena priorities list on the whiteboard15:29
yoctozepto++15:29
mgoddardis there anything in wallaby leftovers we should move across?15:29
yoctozeptomgoddard: proxysql? letsencrypt?15:30
mgoddardi.e. someone available to work on it in the next 4 weeks, cores likely to review it15:30
mgoddardsure15:30
mnasiadkafor letsencrypt I posted a change to bump up haproxy to 2.2, so if we could get this properly working in Xena - it would be a very good achievement15:31
yoctozeptomnasiadka: ++15:31
yoctozeptowe could go straight best approach I guess?15:32
mgoddardswift may happen15:32
yoctozeptoubuntu 2.0 though15:32
yoctozeptoperhaps there is a ppa to unify?15:32
yoctozeptoswift yeah15:32
mnasiadkayoctozepto: Debian has 2.2 straight out of it's repos, Ubuntu has 2.2 from PPA, CentOS 8S has nfv repo15:32
mnasiadkayoctozepto: I updated the change with PPA usage15:33
mnasiadkaa couple days back I think15:33
mgoddardheadphoneJames: I know it's hard, but realistically, which of your features would you prefer to merge in Xena15:33
mgoddard?15:34
yoctozeptomnasiadka: ah, I did not notice, great!15:34
mgoddardrealistically we are unlikely to see some huge surge in reviewer time in the next 4 weeks15:34
mgoddardbut we could try to ignore non-critical bug fixes an maintenance for a while15:35
headphoneJamesmgoddard: preferably the system scope one15:35
mgoddardack15:35
yoctozeptook15:35
mnasiadkaI'll work on the Ansible bump and moving kolla-toolbox to ansible-core/base/however it's called - unless somebody is working on that already.15:37
mgoddarduseful maintenance15:37
mgoddardI think that looks like all the likely candidates from wallaby leftovers15:38
mgoddardare there other features that are not on there?15:38
yoctozepto++, I had hopes for podman but it stalled15:38
yoctozeptoremind me please15:38
yoctozeptoare we going to go ubuntu 22.04 with yoga?15:38
mgoddard4 cycles after bionic, whenever that was15:39
mnasiadkaI think so, doing that now is a bit risky I guess.15:39
mnasiadkaah, 202215:39
mnasiadkaApril15:39
mgoddardit's a bit impossible given it's 21.08 :D15:39
yoctozeptoyeah, I was wondering what mnasiadka answered to15:40
yoctozeptoanyhow, if yoga is going to target 22.04 finally15:40
mnasiadkayoctozepto: I misread 2022 as 2021 :D15:40
yoctozeptothen working on podman could be a lovely priority15:40
yoctozeptoas all the platforms will support it natively15:40
mnasiadkabut this is a discussion for Yoga PTG I guess :)15:40
yoctozeptoyou know me, I love to digress :D15:41
mgoddardtiming could be tight, yoga release is probably in 22.0415:41
yoctozeptoyeah15:41
yoctozeptolemme check when we switched previously15:41
mnasiadkaok then, do we have any kayobe features planned for Xena?15:41
yoctozeptoin vicky15:41
yoctozeptoso october15:42
yoctozeptoso Z15:42
mnasiadkayes, I remember doing that, and the crazy dance with passing ansible_python_interpreter 15:42
yoctozeptoI want to kill that variable but it dropped to lo prio15:42
mnasiadkaWe can discuss that feature on the PTG :)15:42
yoctozeptoit's doable but requires some effort15:43
yoctozepto:-)15:43
yoctozeptolet's move on15:43
yoctozeptowhiteboard cleaned up15:43
yoctozeptothanks mgoddard for driving this mostly15:43
mgoddardadded ceph rgw to the list15:43
yoctozeptonice15:43
mgoddardmgoddard was searching for perfection with priorities, but I think this approach is ok15:44
yoctozeptothanks for appreciating my lower standards in some cases :-) 15:45
opendevreviewMark Goddard proposed openstack/kolla-ansible master: Add kolla-ansible gather-facts command  https://review.opendev.org/c/openstack/kolla-ansible/+/79461015:45
mgoddard^ found a merge conflict while looking through my patches :)15:46
mnasiadkaso, clean up done, lower standards accepted, do we have anything else for Xena priorities?15:46
mgoddardI could add pages of patches :)15:46
mnasiadkaI don't know if scrolling through pages of patches is what we want to do - but we can discuss this approach :)15:47
mgoddardindeed15:47
mgoddardI might dump a list of XS patches15:48
mgoddardas I'm tired of looking at them 15:48
yoctozeptodo a festival of XS patches review?15:48
yoctozeptolike some other teams do?15:48
mgoddardthat would be nice15:48
yoctozepto1 hour a week on video call?15:48
mnasiadkawe could just add a link in the whiteboard and encourage other community members to review them15:48
mnasiadkayoctozepto: reviving Kolla Kall?15:48
yoctozepto(to joke about the quality)15:48
yoctozeptomnasiadka: kind of, but specifically for cores to handle this stuff and enjoy each other15:49
mnasiadkaWe can try, anything new might work for a couple of times ;)15:49
mnasiadkaok, we have two topics left for 10 minutes15:50
mnasiadka#topic handling of APT keys - https://review.opendev.org/c/openstack/kolla/+/784923 (hrw, Fl1nt)15:50
mnasiadkahrw, Fl1nt: stage is yours15:51
yoctozeptomnasiadka: it's gating now15:51
mnasiadkaah, so no stage, unless it breaks CI :)15:51
yoctozeptoI gave +2 on ps2615:52
yoctozeptoand only reno changed a bit15:52
yoctozeptoso re-gave +215:52
yoctozeptoand approved15:52
mnasiadkagood work15:52
yoctozeptominor nit on reno if someone cares about commas15:52
yoctozeptoI'm glad it gets merged15:52
mnasiadkawe'll fix those renos again before release15:53
mnasiadka#topic Yoga PTG planning15:53
yoctozepto(for the next meeting, I will want to discuss some debian stuff)15:53
mnasiadkaSo, PTG is somewhere in October  (18th-22nd)15:54
yoctozeptoyeah15:54
mnasiadkaI'll create the usual etherpads, what's the other usual drill?15:55
mgoddardwe have some slots already15:55
mgoddardso mostly we need to decide on agenda & discussion topics15:55
yoctozepto++15:55
yoctozeptoand we do that in the etherpad15:55
mgoddard#link https://etherpad.opendev.org/p/kolla-yoga-ptg15:55
mgoddardnot populated15:56
yoctozeptoI suggest restoring Xena one on this and going from there15:56
yoctozeptoaction mnasiadka to populate the etherpad?15:56
mnasiadka#action mnasiadka to populate the etherpad15:57
yoctozeptobtw, I don't know if you noticed15:57
mnasiadkaI'll copy paste the Xena one, clean it up/rework and we'll have a solid foundation\15:58
yoctozeptobut Monasca went leaderless (again?)15:58
mnasiadkaI'm afraid Monasca doesn't have a bright future.15:58
yoctozeptome too15:58
yoctozeptoI'm mostly propagating this to you because you are deploying it15:58
mnasiadkaThis is changing, so we might rethink if we need that CI coverage (for the job that is failing all the time).15:59
dtantsurhi folks! I'm struggling to understand if this https://zuul.opendev.org/t/openstack/build/eb452f5af8024837a35e92a5ab8e8f1f/log/primary/logs/ansible/deploy-bifrost#12032 is caused by my Bifrost change or is a breakage in Kolla proper. Could someone give me a hint?15:59
mnasiadkaok, the meeting time is up, dtantsur thanks for joining for the last minute ;-)16:00
dtantsurooops :)16:00
mgoddardthanks mnasiadka 16:00
mnasiadkaIf anybody has any open discussion points - let's continue after the meeting16:00
yoctozeptothanks mnasiadka16:00
mnasiadka#endmeeting kolla16:00
opendevmeetMeeting ended Wed Aug 25 16:00:27 2021 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:00
opendevmeetMinutes:        https://meetings.opendev.org/meetings/kolla/2021/kolla.2021-08-25-15.00.html16:00
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/kolla/2021/kolla.2021-08-25-15.00.txt16:00
opendevmeetLog:            https://meetings.opendev.org/meetings/kolla/2021/kolla.2021-08-25-15.00.log.html16:00
yoctozeptodtantsur: checking16:00
headphoneJamesthanks mnasiadka !!16:00
dtantsuryeah, sorry for interrupting your party :)16:00
dtantsurthe patch in question is https://review.opendev.org/c/openstack/bifrost/+/80600116:01
mnasiadkadtantsur: it doesn't look like it's related to the patch...16:01
yoctozepto"/home/ironic", interesting16:02
dtantsurI've got the same impression16:02
knockknockIs https://bugs.launchpad.net/kolla-ansible/ the appropriate place to report a bug related to kolla binary container images? Found a bug in binary thats not in source16:02
mnasiadkaI don't think we're creating ironic user in bifrost container16:03
mgoddardyoctozepto: I would guess it's due to that /var/lib/<user> change16:03
mnasiadkaah, might be related16:03
mnasiadkaknockknock: https://bugs.launchpad.net/kolla/ is for container images16:03
mgoddardyoctozepto: one of those patches where I thought about saying it may not be a good idea, but I didn't, and later regret it :)16:03
yoctozeptomgoddard: yeah, that could be it, did not it get triggered from kolla?16:03
yoctozeptomnasiadka: knockknock?16:03
yoctozeptoah, it's a username16:04
yoctozeptosorry16:04
knockknockthanks16:04
mnasiadkamgoddard: I think I raised that point in the review :)16:04
mnasiadkaor some related review16:04
*** rpittau is now known as rpittau|afk16:04
yoctozeptolet's revert16:04
mnasiadkain this one, it was written /var/lib/<user> is the default anyway16:04
mnasiadkaso maybe it isn't ;)16:04
yoctozeptoneed bifrost testing on kolla changes16:05
yoctozeptosomeone restore the meeting and action someone on that ;d16:05
yoctozeptodtantsur: we need to fix the job; please feel free to proceed for now16:05
mnasiadkayoctozepto: good idea, just add it to the whiteboard - do we have some CI Ideas section? ;)16:05
dtantsuryoctozepto: ack, thank yoiu16:05
yoctozeptomnasiadka: we have "all CI messy stuff" section16:06
yoctozeptoso it fits16:06
opendevreviewRadosław Piliszek proposed openstack/kolla master: Revert "Config: Set default home to /var/lib/{user}"  https://review.opendev.org/c/openstack/kolla/+/80590016:07
yoctozeptomgoddard, mnasiadka: let's merge ^16:08
yoctozeptoI blocked the related change to first test bifrost with that as well16:09
yoctozeptothe previous patches should be safe16:09
dtantsuryoctozepto: configuring your IRC client to highlight known user names if very handy. there is a weechat extensions for that.16:09
yoctozeptodtantsur: yeah, mine colours nicks but it chose a not-really-colourful colour for that folk :D16:10
yoctozeptoand I did not notice the query16:10
dtantsuryep, can happen16:10
yoctozeptoso went "what the..." mode16:10
dtantsurI've learned that anything can be a nick during my years here! :)16:11
yoctozeptothat true16:11
* dtantsur now stops distracting people from the pleasure of CI debugging16:11
yoctozepto:-)16:12
yoctozeptorevert proposed already, worry not16:12
mgoddardcycle highlights: https://review.opendev.org/c/openstack/releases/+/80602116:20
mgoddardcycle lowlights: https://review.opendev.org/c/openstack/releases/+/806021216:21
opendevreviewMark Goddard proposed openstack/kolla-ansible master: service-ks-register: ensure all user roles exist  https://review.opendev.org/c/openstack/kolla-ansible/+/77445116:23
opendevreviewMark Goddard proposed openstack/kolla-ansible master: Support creating roles and assigning to service users  https://review.opendev.org/c/openstack/kolla-ansible/+/77445216:23
opendevreviewMark Goddard proposed openstack/kolla-ansible master: docs: Add placeholder page for CI & testing information  https://review.opendev.org/c/openstack/kolla-ansible/+/77556416:25
opendevreviewMark Goddard proposed openstack/kolla-ansible master: Performance: Split out prometheus node exporter and cadvisor  https://review.opendev.org/c/openstack/kolla-ansible/+/74008216:27
opendevreviewJames Kirsch proposed openstack/kolla-ansible master: Support for keystone scoped authorization  https://review.opendev.org/c/openstack/kolla-ansible/+/69217916:36
mnasiadkayoctozepto: messy +1 ;)17:00
opendevreviewMerged openstack/kolla master: Debian/Ubuntu: handle APT keys in proper way  https://review.opendev.org/c/openstack/kolla/+/78492317:18
yoctozeptomgoddard: lowlights point nowhere ;-)17:21
mnasiadkayoctozepto: that might be the goal of lowlights ;)17:24
yoctozeptomnasiadka: I did not understand the joke and demand an immediate explanation! :D17:25
yoctozeptomnasiadka: https://review.opendev.org/c/openstack/kolla/+/80590017:42
yoctozeptomnasiadka: let's merge that to rebuild tomorrow17:42
opendevreviewRadosław Piliszek proposed openstack/kolla-ansible master: [DNM] Test bifrost  https://review.opendev.org/c/openstack/kolla-ansible/+/80604018:22
opendevreviewRadosław Piliszek proposed openstack/kolla-ansible master: [DNM] Test bifrost  https://review.opendev.org/c/openstack/kolla-ansible/+/80604018:49
opendevreviewMerged openstack/kolla-ansible master: docs: Add placeholder page for CI & testing information  https://review.opendev.org/c/openstack/kolla-ansible/+/77556418:51
opendevreviewMerged openstack/kolla master: Revert "Config: Set default home to /var/lib/{user}"  https://review.opendev.org/c/openstack/kolla/+/80590020:34

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!