ignaziocassano | Hello, sorry if I ask but I am new on docker and kolla. Is there a mode to show and pull wallaby image version ? In other words....I know every week new source images are published. If I want to show and pull the images 2 weeks old .... | 06:42 |
---|---|---|
yoctozepto | ignaziocassano: those old are gone | 07:00 |
ignaziocassano | So, If I want to simulate an update I must wait next week ? | 07:02 |
ignaziocassano | yoctozepto: anycase I discovered that update stops vm because nova_compute lost connection with libvirtd | 07:03 |
ignaziocassano | yoctozepto: I do not know if your are reading my emails to openstack discussion list | 07:04 |
parallax | ignaziocassano: you could build the images yourself using kolla-build; otherwise, you're right, tags are getting overwritten every week | 07:05 |
parallax | I always thought this is an anti-pattern in a Docker world | 07:06 |
*** rpittau|afk is now known as rpittau | 07:06 | |
ignaziocassano | parallax: Ok thanks, but the problem form me is solving the issue I discussed above. | 07:08 |
ignaziocassano | I am new in kolla. I am testing it. At this time I am using installation made with my ansible playbooks. I never seen situation where vm go doen because libvirtd restars | 07:10 |
ignaziocassano | But it kolla wallaby this happens every time I must update and libvirtd container needs to restart | 07:11 |
ignaziocassano | I do not know if there is a solution. I can image a workaround, updating one compute node at time e moving vm | 07:13 |
ignaziocassano | In emailing list someone wrote: systemd-machined previously did not work inside the contaiener so livert fell back to its generic cgroups backend, ignoring machined managing mancines without it but recently systemd-machined started working in the contianer. | 07:16 |
yoctozepto | ignaziocassano: yeah, I've read those but I'm out of ideas for now :-( | 07:28 |
hrw | morning | 07:30 |
opendevreview | Piotr Parczewski proposed openstack/kolla-ansible master: Add support for configuring Elasticsearch replication https://review.opendev.org/c/openstack/kolla-ansible/+/805954 | 07:30 |
ignaziocassano | yoctozepto: I am reinstalling an then I will check the libvitd container to find if the bug is the same that Sean wrote | 07:30 |
Fl1nt | morning hrw ! | 07:35 |
*** mgoddard- is now known as mgoddard | 07:47 | |
hrw | mgoddard: do we plan to merge apt keys change https://review.opendev.org/c/openstack/kolla/+/784923 or should it go into same bag as infra stuff? | 07:50 |
mnasiadka | morning | 08:06 |
mnasiadka | hrw: we have a bag for that stuff? :) | 08:07 |
hrw | mnasiadka: it is called 'abandonware' | 08:10 |
mnasiadka | hrw: true, around the keys change - shouldn't we update https://docs.openstack.org/kolla/latest/admin/image-building.html#debian-ubuntu ? | 08:12 |
hrw | mnasiadka: it is still valid | 08:14 |
mnasiadka | hrw: but there's no mention of how to override the keys? | 08:15 |
hrw | never was | 08:15 |
hrw | we never said how to override apt keys | 08:16 |
hrw | so if someone overrode them then they figured it out | 08:16 |
hrw | so they will figure out again | 08:16 |
mnasiadka | great user experience | 08:18 |
hrw | apt keys need to be altered only if you add own repos with own keys. which probably no one does otherwise someone would provide some improvements there | 08:22 |
hrw | If you use copy of repos then same keys are in use. Or you follow CI way and disable checks | 08:22 |
mnasiadka | ok, so we just keep that option in the dark then, until somebody comes and complains/improves that | 08:23 |
Fl1nt | hrw, I do | 08:27 |
Fl1nt | For now I still use template-override to do it as I didn't had time to patch it upstream. | 08:28 |
hrw | Fl1nt: can you share override? | 08:28 |
hrw | I am curious of change | 08:28 |
Fl1nt | On my own I simply zeroed the keys arrays. | 08:29 |
Fl1nt | don't check them at all | 08:29 |
Fl1nt | because when you're using offline mode, the only keys that are valid are the default one and the base one | 08:30 |
Fl1nt | if you let say activate an extra repo | 08:30 |
hrw | Fl1nt: your offline repos have regenerated indexes signed with other key? | 08:31 |
Fl1nt | yes | 08:31 |
Fl1nt | well, depends, we have both | 08:31 |
hrw | Fl1nt: and then key added to apt keys or inside of base image? | 08:31 |
Fl1nt | disabled | 08:32 |
hrw | so no key checks at all? like we do on ci? | 08:32 |
Fl1nt | yep | 08:32 |
Fl1nt | GPG Keys management is a pain in the ass when you're offline | 08:32 |
Fl1nt | like | 08:32 |
Fl1nt | here is an exemple | 08:32 |
Fl1nt | from CentOS | 08:32 |
Fl1nt | When you install BaseOS, the GPG Key is correct as it install on keystore | 08:33 |
Fl1nt | however | 08:33 |
hrw | Fl1nt: so for your use it would be good to move whole apt keys stuff to base_debuntu_apt_keys block so you could override it with empty one? | 08:34 |
Fl1nt | once you want to add new repository the "correct" way by using dnf install centos-openstack-release-x or centos-release-epel etc it won't work | 08:34 |
hrw | ah. it is in base_ubuntu_package_key_installation already ;d | 08:34 |
hrw | Fl1nt: override this block with empty | 08:34 |
mgoddard | hrw: can you add it to today's meeting agenda? | 08:34 |
Fl1nt | Yeah, that's how I do it now | 08:34 |
hrw | then no apt keys are fetched and you do not have to empty lists | 08:34 |
hrw | mgoddard: ok | 08:35 |
Fl1nt | base_<os>_key block are zeroed on template-override. | 08:35 |
hrw | mgoddard: the problem is that I may not be present during meeting :( | 08:35 |
Fl1nt | I'll be there today ^^ | 08:35 |
Fl1nt | for once... | 08:36 |
hrw | added | 08:37 |
hrw | Fl1nt: centos-*-release* packages iirc provide keys inside so if you have a way of installing them offline then you have keys | 08:37 |
Fl1nt | Are you sure? | 08:39 |
Fl1nt | I'll recheck | 08:39 |
Fl1nt | but last time I tried it, it wasn't providing them and we had this circle loop where we install them but then need an extra step to sed the baseURL in order to get them correctly working and to correctly retrieve the RPM-GPG key. | 08:40 |
opendevreview | Mark Goddard proposed openstack/kolla-ansible stable/train: DNM: test CentOS on Train https://review.opendev.org/c/openstack/kolla-ansible/+/805971 | 08:41 |
hrw | Fl1nt: https://koji.mbox.centos.org/koji/rpminfo?rpmID=307315 | 08:43 |
hrw | TASK [tensorflow : build TensorFlow 2.6.0-rc2 for cp39-cp39] | 08:49 |
hrw | op | 08:49 |
hrw | s | 08:49 |
*** ykarel is now known as ykarel|lunch | 08:52 | |
ignaziocassano | Hello, anyone tried on wallaby to restart nova_libvirt container ? All vm on the nodes stop to responding and docker exec -it nova_libvirt machinectl list reports "no machines". This is what happens also when and update is done because nova_libvirt is restarted. | 08:59 |
hrw | ignaziocassano: shouldn't some kind of 'nova shutdownnode' be called first? | 09:01 |
hrw | I would treat container restart same as baremetal restart | 09:01 |
hrw | but I do not manage deployments | 09:02 |
ignaziocassano | hrw: I have only restart the nova libvirt container and nova compute logs: https://paste.openstack.org/show/808301/ | 09:04 |
ignaziocassano | this is because when I update the wallaby images and I run a deploy, I lost connection for all vm | 09:05 |
yoctozepto | ignaziocassano: hmm, so you are specifically hit by the libvirt container restart... interesting | 09:08 |
yoctozepto | I have something you may want to try out | 09:08 |
ignaziocassano | yoctozepto: I can test....let me know | 09:08 |
yoctozepto | https://review.opendev.org/c/openstack/kolla-ansible/+/794262 | 09:09 |
yoctozepto | you need to patch your kolla ansible with that | 09:09 |
yoctozepto | then run it and then try booting instances and restarting libvirt | 09:10 |
ignaziocassano | yoctozepto: I do not think is the bug wrote by Seean Mooney because machinectl list show running machines running it in the container and also out of container | 09:10 |
yoctozepto | note, after applying this patch, machinectl should be unable to list the instances all the time | 09:10 |
yoctozepto | ignaziocassano: well, sure, but I'm just giving you a "more isolation" patch which prevents machinectl from knowing at all | 09:10 |
ignaziocassano | yoctozepto: I must apply the above patch and then run a deploy ? | 09:12 |
hrw | 'nova host-evacuate-live HOSTNAME', then restart/update and add it back to computes? | 09:12 |
ignaziocassano | <hwr> this could be a woraround. It works but we must try to solve this problem | 09:16 |
opendevreview | Mark Goddard proposed openstack/kolla-ansible master: nova: don't fail all hosts on service registration failures https://review.opendev.org/c/openstack/kolla-ansible/+/805599 | 09:19 |
ignaziocassano | yoctozepto: I must apply the above patch and then run a deploy ? | 09:20 |
yoctozepto | ignaziocassano: yes | 09:23 |
yoctozepto | after deploy it should change the behaviour | 09:23 |
yoctozepto | you should be able to spin up an instance AND *not* see it in machinectl | 09:24 |
yoctozepto | then try restarting libvirt | 09:24 |
yoctozepto | if you still see the machine in machinectl then the patch was not properly applied or I'm missing something else :D | 09:24 |
ignaziocassano | yoctozepto: I am running the deploy | 09:24 |
ignaziocassano | yoctozepto: after deployng with the patch hypervisors are marked down and become unresponsive. Trying to connect via ssh take a long time. | 09:45 |
yoctozepto | ignaziocassano: well, the hypervisors themselves should be fine with this patch; at least that was tested ;-) | 09:48 |
ignaziocassano | yoctozepto: libvirt.log reports: mounting "/var/lib/nova/mnt" to rootfs at "/var/lib/nova/mnt" caused: mount through procfd: no space left on device: unknown | 09:53 |
ignaziocassano | but on var I have a lot of space | 09:54 |
ignaziocassano | var and / ave a lot of free space.....any help ? | 10:01 |
ignaziocassano | yoctozepto: I am new on docker , please, why libvirtd report no space if /var have a lot of space available ? | 10:12 |
ignaziocassano | pwd | 10:13 |
opendevreview | Merged openstack/kolla-ansible stable/wallaby: Remove an unused file https://review.opendev.org/c/openstack/kolla-ansible/+/805814 | 10:15 |
opendevreview | Merged openstack/kolla-ansible stable/victoria: Remove an unused file https://review.opendev.org/c/openstack/kolla-ansible/+/805815 | 10:15 |
opendevreview | Merged openstack/kolla-ansible stable/ussuri: Remove an unused file https://review.opendev.org/c/openstack/kolla-ansible/+/805816 | 10:15 |
*** ykarel|lunch is now known as ykarel | 10:17 | |
amoralej | hi hrw, wrt https://github.com/openstack/kolla/commit/383364e671fb82e3c0ddca37a4690a1a33be3b54 | 10:25 |
hrw | amoralej: yes? | 10:27 |
amoralej | this is breaking kolla in rdo :) | 10:33 |
amoralej | as we use jinja package from centos | 10:33 |
amoralej | and both centos stream 8 and 9 ships version 2 | 10:33 |
amoralej | and we prefer to not duplicate packages provided in centos | 10:34 |
hrw | xena openstack/requirements lists jinja 3.0.1 now | 10:34 |
amoralej | yeah but, iiuc, jinja3 still can work with 2 syntax? | 10:35 |
amoralej | i mean, other projects still work with >=2.x | 10:35 |
yoctozepto | ignaziocassano: interesting, I have not met this issue before | 10:35 |
amoralej | but i understand are being tested with 3.0.1 | 10:35 |
amoralej | as it's in u-c | 10:35 |
hrw | amoralej: 3.0.1 does. 3.1 will not | 10:35 |
hrw | amoralej: so feel free to propose revert for Xena and then we can go back to it in Yeti. | 10:36 |
amoralej | so by 3.1 timeframe it will force users to update templates to 3.x specific format? | 10:36 |
hrw | amoralej: no - contextfunction -> pass_context | 10:36 |
hrw | amoralej: templates do not change. custom template functions need update | 10:37 |
ignaziocassano | yoctozepto: I googled for lookinf if other users are affected by this problem. I found one only with no answers. I am updating packages to verify if it solves | 10:37 |
hrw | amoralej: https://jinja.palletsprojects.com/en/3.0.x/api/#jinja2.contextfunction | 10:37 |
hrw | amoralej: https://review.opendev.org/c/openstack/kolla/+/798263 is what you want to revert then | 10:40 |
amoralej | hrw, thanks for that info, so, it seems currently only renderspec is using it https://codesearch.opendev.org/?q=contextfunction&i=nope&literal=nope&files=&excludeFiles=&repos= | 10:41 |
amoralej | hrw, you think kolla users working on rdo are using kolla rpm to build containers? | 10:42 |
amoralej | or they just use the containers built by kolla proyect? | 10:42 |
hrw | amoralej: no idea to be honest | 10:42 |
hrw | amoralej: we usually suggest to follow git branches to build stable/* containers | 10:43 |
amoralej | i was thinking if the package is being used by users... | 10:43 |
amoralej | in the past was dependency for tripleo, but not now | 10:43 |
amoralej | said this, we want to stay as a supported and friendly project for kolla | 10:43 |
hrw | amoralej: do you package airship? | 10:44 |
amoralej | nop | 10:44 |
amoralej | but i'm not sure if airship uses rdo | 10:45 |
amoralej | we didn't have much interaction with airship project tbh | 10:45 |
hrw | ok. then propose revert of jinja bump with centos repo info and I think that we can stay on jinja2 2.x for Xena | 10:45 |
hrw | I did not know about airship 2 minutes ago ;D | 10:45 |
amoralej | :) | 10:45 |
amoralej | ykarel, ^ | 10:46 |
amoralej | thanks hrw | 10:46 |
ykarel | Thanks amoralej hrw | 10:48 |
opendevreview | Alfredo Moralejo proposed openstack/kolla master: Revert "Update code to Jinja 3.0 style" https://review.opendev.org/c/openstack/kolla/+/805897 | 10:50 |
hrw | +2 | 10:51 |
Fl1nt | amoralej, I build kolla containers from pip with our RDO Centos based platform if it can help | 11:02 |
amoralej | thanks Fl1nt, i wonder if that's the usual users pattern | 11:03 |
amoralej | where is documented how to build containers? | 11:03 |
Fl1nt | usual user pattern if you're on enterprise context | 11:03 |
Fl1nt | and so you can't use kolla created images from docker hub | 11:03 |
Fl1nt | amoralej, you mean this doc? https://docs.openstack.org/kolla/latest/admin/image-building.html | 11:04 |
amoralej | yep | 11:04 |
amoralej | so, no mention to intall kolla-build from rpm package | 11:05 |
Fl1nt | just for you to know, at least at Ubisoft, Blizzard and Govs we use the following process: | 11:05 |
Fl1nt | pip install kolla == version | 11:05 |
Fl1nt | then use kolla-build from there | 11:05 |
amoralej | good to know, my question is if we would break someone by not shiping kolla package in rdo repo | 11:06 |
amoralej | anyway, i'd prefer to keep it | 11:06 |
Fl1nt | I can only give you feedback on my experience, but it's not representative of the whole world so I guess it's better to keep it ^^ | 11:07 |
amoralej | yep, i agree | 11:10 |
amoralej | thanks Fl1nt anyway! | 11:10 |
Fl1nt | sure you're welcome | 11:12 |
ignaziocassano | yoctozepto: the patch seems not working. I starded with a cleaned environment. an deploy does nt work : FAILED - RETRYING: Waiting for nova-compute services to register themselves (1 retries left). | 12:04 |
ignaziocassano | yoctozepto: nova_libvirt does not start and does not write anything in logs | 12:07 |
*** amoralej is now known as amoralej|lunch | 12:34 | |
yoctozepto | ignaziocassano: are you sure you have applied it correctly? | 12:42 |
yoctozepto | because I know it works; I was only wondering if it fixes your issue, not if it works | 12:42 |
ignaziocassano | yoctozepto: only one question to check if I am using the correct workflow: docker is installed on controllers and compute nodes by boostrap and/or prechecks ? | 12:44 |
yoctozepto | ignaziocassano: yes, by bootstrap; you could also install it manually; the bootstrap is just a convenience ;-) | 12:51 |
ignaziocassano | I will try again to clean my anvironment and deploy again with your patch | 12:52 |
ignaziocassano | right ? | 12:52 |
yoctozepto | ignaziocassano: you should not need to clean everything up though | 12:56 |
yoctozepto | but you can try | 12:56 |
yoctozepto | stackhpc heads up ~> monasca leaderless https://etherpad.opendev.org/p/yoga-leaderless | 12:56 |
*** amoralej|lunch is now known as amoralej | 13:01 | |
opendevreview | Piotr Parczewski proposed openstack/kolla master: Fix Elasticsearch curator https://review.opendev.org/c/openstack/kolla/+/805990 | 13:04 |
ignaziocassano | yoctozepto: libvirt container does not start : OSError: [Errno 16] Device or resource busy: '/etc/libvirt/secrets' | 13:13 |
yoctozepto | ignaziocassano: you must have misapplied the patch :-( | 13:30 |
ignaziocassano | yoctozepto: I downloaded it and I copied it under /usr/local/share/kolla-ansible/ansible/roles/nova-cell/defaults | 13:43 |
ignaziocassano | yoctozepto: I moved main.yml in main.yml.orig and substitued it with the main.yml I downloaded | 13:45 |
yoctozepto | ignaziocassano: ah, yeah, that will surely break | 13:49 |
yoctozepto | I will guide you then | 13:49 |
yoctozepto | restore your original file | 13:49 |
yoctozepto | go to the first line that says | 13:50 |
yoctozepto | - "/run/:/run/:shared" | 13:50 |
yoctozepto | and replace it with | 13:50 |
yoctozepto | - "/run/openvswitch:/run/openvswitch" | 13:50 |
yoctozepto | go to the second line that says | 13:50 |
yoctozepto | - "/run/:/run/:shared" | 13:51 |
yoctozepto | and remove this one | 13:51 |
yoctozepto | sorry, the second line actually has less slashes: | 13:52 |
yoctozepto | - "/run:/run:shared" | 13:52 |
yoctozepto | but otherwise the instructions stay the same | 13:52 |
ignaziocassano | yoctozepto: that's all ? | 13:57 |
yoctozepto | yes, two changes needed to be done | 13:59 |
yoctozepto | you can now diff your original with your modified original to make sure only that has happened | 13:59 |
yoctozepto | and redo your deployment | 14:00 |
ignaziocassano | yoctozepto: let me to finish the deployment without patch. Then I'll apply it following your istruction. It is only for be sure it is working | 14:02 |
yoctozepto | ignaziocassano: yes; libvirt should still be fine after this patch | 14:02 |
yoctozepto | this patch may help with machinectl meddling | 14:02 |
yoctozepto | but this is what we are testing :-) | 14:02 |
jd2021 | Hi | 14:21 |
ignaziocassano | yoctozepto: https://paste.openstack.org/show/808314/ | 14:21 |
jd2021 | We are trying to use kolla-ansible but we have encountered some issues related with the storage backend. | 14:22 |
jd2021 | Our storage is a Netapp e-series, which had an cinder driver but it has been dropped in last Openstack releases | 14:23 |
jd2021 | Now, it seems that we can only use LVM instead | 14:24 |
jd2021 | however, the Kolla documentation is not clear. It refers two approachs, the LVM backend and the LMV2/iscsi backend. | 14:24 |
jd2021 | the documentation only says that to activate either one of these two backend, to ebable enable_cinder_backend_lvm: "yes" | 14:26 |
yoctozepto | ignaziocassano: and also another line to remove | 14:28 |
jd2021 | but when doing that, it tries to use iscsi to connect to a cinder storage node, which is not the storage per se, and then connect the compute nodes with the storage node via iscsi. | 14:28 |
ignaziocassano | jd202 | 14:28 |
ignaziocassano | yoctozepto: but if download the patch file and I execute the patch command is not more easy | 14:29 |
jd2021 | but to have the openstack storage node with a volume, it must be configured through iscsi to the netapp appliance (iscsi => lvm => iscsi) | 14:29 |
opendevreview | Piotr Parczewski proposed openstack/kolla master: Fix Elasticsearch Curator https://review.opendev.org/c/openstack/kolla/+/805990 | 14:30 |
jd2021 | it seems that the correct way is to just configure the same lvm volume backed by iscsi to the netapp appliance on all the compute nodes, but the kolla config does not seams to have this option available | 14:30 |
yoctozepto | ignaziocassano: yeah, that would be easier but you told me you did not use patch but replaced the files | 14:30 |
jd2021 | I would be very grateful if someone could shed some light onto this: How to use static block volumes via LVM directly on the compute nodes? | 14:32 |
ignaziocassano | yoctozepto: you are right. Now I download the patch file and I apply the patch. Sicne I must go out of office, I 'll contct you tomorrow | 14:33 |
yoctozepto | ignaziocassano: ok, see you | 14:34 |
ignaziocassano | yoctozepto>: please chek if the patch is applyed well: https://paste.openstack.org/show/808315/ | 14:39 |
*** hlm is now known as Guest5377 | 14:42 | |
yoctozepto | ignaziocassano: seems so! | 14:43 |
jd2021 | Can anyone? | 14:45 |
jd2021 | If this is not the right channel for this kind of questions, can someone point to the right one? | 14:45 |
jd2021 | Can anyone help? | 14:46 |
ignaziocassano | yoctozepto: see you tomorrow. Meanwhile is anything changed for cinder. Last week the volume service was active on all controllers. Now only one the first | 14:49 |
mnasiadka | jd2021: you mean that you want to expose a big disk from netapp to your storage host, and then use lvm/iscsi on the clients? | 14:49 |
mnasiadka | mgoddard mnasiadka hrw egonzalez yoctozepto rafaelweingartne cosmicsound osmanlicilegi bbezak parallax Fl1nt | 14:50 |
mnasiadka | meeting in 10 | 14:50 |
ignaziocassano | jd2021: am using netapp but with nfs protocol | 14:51 |
jd2021 | I would like to avoid using iscsi to connect the nova nodes to an intermediate node, which in turn connects to the netapp. | 14:52 |
jd2021 | the e-series only supports iscsi, not nfs | 14:52 |
mnasiadka | jd2021: you could just use iscsi to mount /var/lib/nova on the compute hosts, and use this for ephemeral storage - but for cinder I think the only option is the intermediate path | 14:53 |
jd2021 | the desided solution is to have a single iscsi connection from the clients (nova node) to the netapp | 14:53 |
ignaziocassano | jd2021: so I think you must provide a cinder.conf on /etc/kolla/config | 14:53 |
mnasiadka | jd2021: but you could confirm this with the cinder folks on #openstack-cinder | 14:53 |
jd2021 | I see. I'll check. But just to be clear, what is the difference between the LVM backend and LVM2/ISCSI options on kolla, since both use the same variable to be enabled? | 14:55 |
ignaziocassano | kolla-ansible -i /etc/kolla/multinode -e ansible_u | 14:55 |
mnasiadka | jd2021: that's the same thing | 14:57 |
jd2021 | ok | 14:57 |
jd2021 | the documentation seams to point as if they were different | 14:58 |
ignaziocassano | jd2021: I do not know If what I writing can help you: I configured iscsi for dell but out of kolla. It ceate a dell volume for each cinder volume and connect it with iscsi using multipath | 14:58 |
mnasiadka | jd2021: kolla-ansible is just a deployment project, docs might be sometimes old and not accurate, but you should always refer to particular project docs - in this case cinder | 14:59 |
mnasiadka | #startmeeting kolla | 15:00 |
opendevmeet | Meeting started Wed Aug 25 15:00:22 2021 UTC and is due to finish in 60 minutes. The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
opendevmeet | The meeting name has been set to 'kolla' | 15:00 |
yoctozepto | \o/ | 15:00 |
mnasiadka | #topic rollcall | 15:00 |
yoctozepto | \o/ | 15:00 |
mnasiadka | o/ | 15:00 |
mgoddard | \o | 15:01 |
headphoneJames | hopefully some system scope o/ | 15:01 |
jd2021 | ignaziocassano: that's what we would like to do (dell volume example). I'll will look into it. ~ | 15:01 |
parallax | o/ | 15:01 |
jd2021 | thanks | 15:02 |
mnasiadka | jd2021: we're holding a meeting now, please follow up on non-meeting topics after it ends :) | 15:03 |
mnasiadka | #topic agenda | 15:03 |
yoctozepto | headphoneJames: system scope? ;o | 15:03 |
mnasiadka | * Announcements | 15:03 |
mnasiadka | ** Congratulations to mnasiadka on becoming PTL :) | 15:03 |
mnasiadka | * Review action items from the last meeting | 15:03 |
mnasiadka | * CI status | 15:03 |
mnasiadka | * Release tasks | 15:03 |
mnasiadka | * Xena cycle planning | 15:03 |
mnasiadka | ** Xena feature prioritisation https://docs.google.com/spreadsheets/d/1BuVMwP8eLnOVJDX8f3Nb6hCrNcNpRQl57T2ENU9Xao8 | 15:03 |
mnasiadka | * handling of APT keys - https://review.opendev.org/c/openstack/kolla/+/784923 (hrw, Fl1nt) | 15:03 |
mnasiadka | * Yoga PTG planning | 15:03 |
ignaziocassano | jd2021: so you must look at cinder driver documentation for netapp model you are using, and the create the cinder.conf for it and put it in /etc/kolla/config/cinder.conf. I think there is also somthing to enable multipath for libvirt | 15:03 |
mnasiadka | * Open discussion | 15:03 |
headphoneJames | congrads to mnasiadka!! | 15:03 |
parallax | :) | 15:04 |
mnasiadka | #topic Announcements | 15:04 |
yoctozepto | congrats to mnasiadka <3 | 15:04 |
mgoddard | +1 | 15:04 |
mnasiadka | Thanks for all the good votes, hoping mgoddard will help during the transition period :-) | 15:05 |
mgoddard | of course! | 15:05 |
* yoctozepto will be there to ensure mgoddard helps | 15:05 | |
yoctozepto | :-) | 15:05 |
ignaziocassano | jd2021: kolla add only some automation to openstack and for configuration of single cinder driver you must look at its documentation | 15:05 |
jd2021 | The cinder netapp driver has dropped the support for the e-series a few years ago (unified driver) | 15:05 |
yoctozepto | ignaziocassano, jd2021: please don't interrupt the meeting | 15:05 |
opendevreview | Merged openstack/kolla master: Revert "Update code to Jinja 3.0 style" https://review.opendev.org/c/openstack/kolla/+/805897 | 15:05 |
ignaziocassano | sorry | 15:05 |
ignaziocassano | good bye | 15:06 |
mnasiadka | Ok then, let's continue | 15:06 |
mnasiadka | #topic Review action items from the last meeting | 15:06 |
jd2021 | sorry, bye | 15:06 |
mnasiadka | seems none outstanding from last time | 15:07 |
mnasiadka | oh, mgoddard to propose cycle highlights | 15:07 |
yoctozepto | he did not :O | 15:08 |
mgoddard | hmm, I think I did not | 15:08 |
mgoddard | action me again | 15:08 |
mnasiadka | #action mgoddard to propose initial cycle highlights | 15:08 |
yoctozepto | mortal stack : action him! | 15:08 |
mnasiadka | did we merge the binary switch to current release? there was some problem with UCA | 15:09 |
yoctozepto | (come on, no mortal kombat players?) | 15:09 |
*** amoralej is now known as amoralej|off | 15:10 | |
mnasiadka | yoctozepto: high school times, nearly forgot about it ;) | 15:10 |
yoctozepto | mnasiadka: :D | 15:10 |
mgoddard | yes, UCA is now Xena | 15:10 |
yoctozepto | I think we merged the relevant patches | 15:10 |
yoctozepto | debian will release when openstack does | 15:10 |
mnasiadka | ok then | 15:11 |
mnasiadka | #topic CI status | 15:11 |
mnasiadka | seems it's green-ish | 15:12 |
mnasiadka | do we need to discuss any particular CI issue? | 15:12 |
mgoddard | I'm not aware of any new ones | 15:12 |
yoctozepto | nope | 15:12 |
yoctozepto | everything known | 15:13 |
mnasiadka | great to hear/see | 15:13 |
mnasiadka | #topic Release tasks | 15:13 |
knockknock | hello | 15:14 |
mnasiadka | R-5 cycle highlights deadline - that we have action on mgoddard | 15:14 |
mgoddard | knockknock: we're in a meeting currently, should end in ~45m | 15:15 |
knockknock | I'm setting up openstack for the first time with kolla and I either misconfigured something or found a bug. It looks like a file either isn't being downloaded or is missing from the docker image. Any suggestions on figuring out if I screwed something up or if it's an actual bug? | 15:15 |
mnasiadka | R-1: Prepare for RC1 & stable branch creation - that will be next | 15:15 |
knockknock | ah sorry | 15:15 |
yoctozepto | how far till feature freeze | 15:15 |
yoctozepto | I might want to drop in support for consul (for masakari) | 15:16 |
mnasiadka | R-5 is feature freeze - so next week | 15:16 |
mnasiadka | should we identify features we want to push in, maybe hashtag them (?) and focus on reviewing? | 15:17 |
mgoddard | crikey, feature freeze next week | 15:17 |
mnasiadka | that's what they say here: https://releases.openstack.org/xena/schedule.html | 15:17 |
mgoddard | ah, that's openstack ff | 15:18 |
mgoddard | we have our own | 15:18 |
mgoddard | I don't think it's included in the timeline | 15:19 |
yoctozepto | mnasiadka: stop scaring poor people | 15:19 |
mgoddard | "There is no official feature freeze date for the cycle-trailing model, but we aim to freeze three weeks after the common feature freeze" | 15:19 |
yoctozepto | mgoddard: when do we want it then? I need ~month I guess? | 15:19 |
mgoddard | R-2 | 15:19 |
mnasiadka | mgoddard: it's not in https://docs.openstack.org/kolla/latest/contributor/release-management.html#release-schedule - should we add it there? | 15:19 |
yoctozepto | then that should be enough + asking for FFE | 15:19 |
yoctozepto | ack | 15:19 |
mgoddard | yes, we should add | 15:19 |
mnasiadka | any volunteer for adding? ;) | 15:20 |
mgoddard | go on then | 15:20 |
mnasiadka | ok then, let me take this, should be easyfix | 15:22 |
mgoddard | I meant I can do it | 15:22 |
mnasiadka | ah | 15:22 |
mnasiadka | #action mgoddard to add FFE on R-2 to release-management docs | 15:23 |
mnasiadka | thank you mgoddard | 15:23 |
opendevreview | Mark Goddard proposed openstack/kolla master: docs: add feature freeze to release management https://review.opendev.org/c/openstack/kolla/+/806013 | 15:23 |
mgoddard | done :D | 15:23 |
mnasiadka | that's the spirit! | 15:23 |
mnasiadka | #topic Xena cycle planning | 15:23 |
mnasiadka | Xena feature prioritisation https://docs.google.com/spreadsheets/d/1BuVMwP8eLnOVJDX8f3Nb6hCrNcNpRQl57T2ENU9Xao8 | 15:23 |
mnasiadka | This has been edited last a week ago by mgoddard, should we work into converting this into a feature list with links to Gerrit topic/changes? Or what's the plan? | 15:24 |
mgoddard | maybe this cycle we should be a bit less rigorous, with only 4 weeks until freeze? | 15:25 |
headphoneJames | Also, in the other direction, how does one propose a feature to the list? | 15:25 |
mgoddard | e.g. if you can think of a feature (or find one in that sheet) that you'd like in Xena, add it to the whiteboard | 15:25 |
mnasiadka | no rigor, surely - but it would be nice for reviewers to have a ,,focus list'' | 15:25 |
yoctozepto | yes, let's not use up too many sources on this part; let's try to clean up the whiteboard to get more focus | 15:26 |
yoctozepto | well, cleanup and more links and status reports | 15:26 |
yoctozepto | as delivered by kevko for proxysql | 15:26 |
mgoddard | headphoneJames: the list is all open patches as of a few weeks ago. it hasn't been massaged yet | 15:26 |
yoctozepto | no massages, no fun | 15:27 |
Fl1nt | sorry, get TZed | 15:28 |
Fl1nt | :( | 15:28 |
mgoddard | I've added some sections to the Xena priorities list on the whiteboard | 15:29 |
yoctozepto | ++ | 15:29 |
mgoddard | is there anything in wallaby leftovers we should move across? | 15:29 |
yoctozepto | mgoddard: proxysql? letsencrypt? | 15:30 |
mgoddard | i.e. someone available to work on it in the next 4 weeks, cores likely to review it | 15:30 |
mgoddard | sure | 15:30 |
mnasiadka | for letsencrypt I posted a change to bump up haproxy to 2.2, so if we could get this properly working in Xena - it would be a very good achievement | 15:31 |
yoctozepto | mnasiadka: ++ | 15:31 |
yoctozepto | we could go straight best approach I guess? | 15:32 |
mgoddard | swift may happen | 15:32 |
yoctozepto | ubuntu 2.0 though | 15:32 |
yoctozepto | perhaps there is a ppa to unify? | 15:32 |
yoctozepto | swift yeah | 15:32 |
mnasiadka | yoctozepto: Debian has 2.2 straight out of it's repos, Ubuntu has 2.2 from PPA, CentOS 8S has nfv repo | 15:32 |
mnasiadka | yoctozepto: I updated the change with PPA usage | 15:33 |
mnasiadka | a couple days back I think | 15:33 |
mgoddard | headphoneJames: I know it's hard, but realistically, which of your features would you prefer to merge in Xena | 15:33 |
mgoddard | ? | 15:34 |
yoctozepto | mnasiadka: ah, I did not notice, great! | 15:34 |
mgoddard | realistically we are unlikely to see some huge surge in reviewer time in the next 4 weeks | 15:34 |
mgoddard | but we could try to ignore non-critical bug fixes an maintenance for a while | 15:35 |
headphoneJames | mgoddard: preferably the system scope one | 15:35 |
mgoddard | ack | 15:35 |
yoctozepto | ok | 15:35 |
mnasiadka | I'll work on the Ansible bump and moving kolla-toolbox to ansible-core/base/however it's called - unless somebody is working on that already. | 15:37 |
mgoddard | useful maintenance | 15:37 |
mgoddard | I think that looks like all the likely candidates from wallaby leftovers | 15:38 |
mgoddard | are there other features that are not on there? | 15:38 |
yoctozepto | ++, I had hopes for podman but it stalled | 15:38 |
yoctozepto | remind me please | 15:38 |
yoctozepto | are we going to go ubuntu 22.04 with yoga? | 15:38 |
mgoddard | 4 cycles after bionic, whenever that was | 15:39 |
mnasiadka | I think so, doing that now is a bit risky I guess. | 15:39 |
mnasiadka | ah, 2022 | 15:39 |
mnasiadka | April | 15:39 |
mgoddard | it's a bit impossible given it's 21.08 :D | 15:39 |
yoctozepto | yeah, I was wondering what mnasiadka answered to | 15:40 |
yoctozepto | anyhow, if yoga is going to target 22.04 finally | 15:40 |
mnasiadka | yoctozepto: I misread 2022 as 2021 :D | 15:40 |
yoctozepto | then working on podman could be a lovely priority | 15:40 |
yoctozepto | as all the platforms will support it natively | 15:40 |
mnasiadka | but this is a discussion for Yoga PTG I guess :) | 15:40 |
yoctozepto | you know me, I love to digress :D | 15:41 |
mgoddard | timing could be tight, yoga release is probably in 22.04 | 15:41 |
yoctozepto | yeah | 15:41 |
yoctozepto | lemme check when we switched previously | 15:41 |
mnasiadka | ok then, do we have any kayobe features planned for Xena? | 15:41 |
yoctozepto | in vicky | 15:41 |
yoctozepto | so october | 15:42 |
yoctozepto | so Z | 15:42 |
mnasiadka | yes, I remember doing that, and the crazy dance with passing ansible_python_interpreter | 15:42 |
yoctozepto | I want to kill that variable but it dropped to lo prio | 15:42 |
mnasiadka | We can discuss that feature on the PTG :) | 15:42 |
yoctozepto | it's doable but requires some effort | 15:43 |
yoctozepto | :-) | 15:43 |
yoctozepto | let's move on | 15:43 |
yoctozepto | whiteboard cleaned up | 15:43 |
yoctozepto | thanks mgoddard for driving this mostly | 15:43 |
mgoddard | added ceph rgw to the list | 15:43 |
yoctozepto | nice | 15:43 |
mgoddard | mgoddard was searching for perfection with priorities, but I think this approach is ok | 15:44 |
yoctozepto | thanks for appreciating my lower standards in some cases :-) | 15:45 |
opendevreview | Mark Goddard proposed openstack/kolla-ansible master: Add kolla-ansible gather-facts command https://review.opendev.org/c/openstack/kolla-ansible/+/794610 | 15:45 |
mgoddard | ^ found a merge conflict while looking through my patches :) | 15:46 |
mnasiadka | so, clean up done, lower standards accepted, do we have anything else for Xena priorities? | 15:46 |
mgoddard | I could add pages of patches :) | 15:46 |
mnasiadka | I don't know if scrolling through pages of patches is what we want to do - but we can discuss this approach :) | 15:47 |
mgoddard | indeed | 15:47 |
mgoddard | I might dump a list of XS patches | 15:48 |
mgoddard | as I'm tired of looking at them | 15:48 |
yoctozepto | do a festival of XS patches review? | 15:48 |
yoctozepto | like some other teams do? | 15:48 |
mgoddard | that would be nice | 15:48 |
yoctozepto | 1 hour a week on video call? | 15:48 |
mnasiadka | we could just add a link in the whiteboard and encourage other community members to review them | 15:48 |
mnasiadka | yoctozepto: reviving Kolla Kall? | 15:48 |
yoctozepto | (to joke about the quality) | 15:48 |
yoctozepto | mnasiadka: kind of, but specifically for cores to handle this stuff and enjoy each other | 15:49 |
mnasiadka | We can try, anything new might work for a couple of times ;) | 15:49 |
mnasiadka | ok, we have two topics left for 10 minutes | 15:50 |
mnasiadka | #topic handling of APT keys - https://review.opendev.org/c/openstack/kolla/+/784923 (hrw, Fl1nt) | 15:50 |
mnasiadka | hrw, Fl1nt: stage is yours | 15:51 |
yoctozepto | mnasiadka: it's gating now | 15:51 |
mnasiadka | ah, so no stage, unless it breaks CI :) | 15:51 |
yoctozepto | I gave +2 on ps26 | 15:52 |
yoctozepto | and only reno changed a bit | 15:52 |
yoctozepto | so re-gave +2 | 15:52 |
yoctozepto | and approved | 15:52 |
mnasiadka | good work | 15:52 |
yoctozepto | minor nit on reno if someone cares about commas | 15:52 |
yoctozepto | I'm glad it gets merged | 15:52 |
mnasiadka | we'll fix those renos again before release | 15:53 |
mnasiadka | #topic Yoga PTG planning | 15:53 |
yoctozepto | (for the next meeting, I will want to discuss some debian stuff) | 15:53 |
mnasiadka | So, PTG is somewhere in October (18th-22nd) | 15:54 |
yoctozepto | yeah | 15:54 |
mnasiadka | I'll create the usual etherpads, what's the other usual drill? | 15:55 |
mgoddard | we have some slots already | 15:55 |
mgoddard | so mostly we need to decide on agenda & discussion topics | 15:55 |
yoctozepto | ++ | 15:55 |
yoctozepto | and we do that in the etherpad | 15:55 |
mgoddard | #link https://etherpad.opendev.org/p/kolla-yoga-ptg | 15:55 |
mgoddard | not populated | 15:56 |
yoctozepto | I suggest restoring Xena one on this and going from there | 15:56 |
yoctozepto | action mnasiadka to populate the etherpad? | 15:56 |
mnasiadka | #action mnasiadka to populate the etherpad | 15:57 |
yoctozepto | btw, I don't know if you noticed | 15:57 |
mnasiadka | I'll copy paste the Xena one, clean it up/rework and we'll have a solid foundation\ | 15:58 |
yoctozepto | but Monasca went leaderless (again?) | 15:58 |
mnasiadka | I'm afraid Monasca doesn't have a bright future. | 15:58 |
yoctozepto | me too | 15:58 |
yoctozepto | I'm mostly propagating this to you because you are deploying it | 15:58 |
mnasiadka | This is changing, so we might rethink if we need that CI coverage (for the job that is failing all the time). | 15:59 |
dtantsur | hi folks! I'm struggling to understand if this https://zuul.opendev.org/t/openstack/build/eb452f5af8024837a35e92a5ab8e8f1f/log/primary/logs/ansible/deploy-bifrost#12032 is caused by my Bifrost change or is a breakage in Kolla proper. Could someone give me a hint? | 15:59 |
mnasiadka | ok, the meeting time is up, dtantsur thanks for joining for the last minute ;-) | 16:00 |
dtantsur | ooops :) | 16:00 |
mgoddard | thanks mnasiadka | 16:00 |
mnasiadka | If anybody has any open discussion points - let's continue after the meeting | 16:00 |
yoctozepto | thanks mnasiadka | 16:00 |
mnasiadka | #endmeeting kolla | 16:00 |
opendevmeet | Meeting ended Wed Aug 25 16:00:27 2021 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:00 |
opendevmeet | Minutes: https://meetings.opendev.org/meetings/kolla/2021/kolla.2021-08-25-15.00.html | 16:00 |
opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/kolla/2021/kolla.2021-08-25-15.00.txt | 16:00 |
opendevmeet | Log: https://meetings.opendev.org/meetings/kolla/2021/kolla.2021-08-25-15.00.log.html | 16:00 |
yoctozepto | dtantsur: checking | 16:00 |
headphoneJames | thanks mnasiadka !! | 16:00 |
dtantsur | yeah, sorry for interrupting your party :) | 16:00 |
dtantsur | the patch in question is https://review.opendev.org/c/openstack/bifrost/+/806001 | 16:01 |
mnasiadka | dtantsur: it doesn't look like it's related to the patch... | 16:01 |
yoctozepto | "/home/ironic", interesting | 16:02 |
dtantsur | I've got the same impression | 16:02 |
knockknock | Is https://bugs.launchpad.net/kolla-ansible/ the appropriate place to report a bug related to kolla binary container images? Found a bug in binary thats not in source | 16:02 |
mnasiadka | I don't think we're creating ironic user in bifrost container | 16:03 |
mgoddard | yoctozepto: I would guess it's due to that /var/lib/<user> change | 16:03 |
mnasiadka | ah, might be related | 16:03 |
mnasiadka | knockknock: https://bugs.launchpad.net/kolla/ is for container images | 16:03 |
mgoddard | yoctozepto: one of those patches where I thought about saying it may not be a good idea, but I didn't, and later regret it :) | 16:03 |
yoctozepto | mgoddard: yeah, that could be it, did not it get triggered from kolla? | 16:03 |
yoctozepto | mnasiadka: knockknock? | 16:03 |
yoctozepto | ah, it's a username | 16:04 |
yoctozepto | sorry | 16:04 |
knockknock | thanks | 16:04 |
mnasiadka | mgoddard: I think I raised that point in the review :) | 16:04 |
mnasiadka | or some related review | 16:04 |
*** rpittau is now known as rpittau|afk | 16:04 | |
yoctozepto | let's revert | 16:04 |
mnasiadka | in this one, it was written /var/lib/<user> is the default anyway | 16:04 |
mnasiadka | so maybe it isn't ;) | 16:04 |
yoctozepto | need bifrost testing on kolla changes | 16:05 |
yoctozepto | someone restore the meeting and action someone on that ;d | 16:05 |
yoctozepto | dtantsur: we need to fix the job; please feel free to proceed for now | 16:05 |
mnasiadka | yoctozepto: good idea, just add it to the whiteboard - do we have some CI Ideas section? ;) | 16:05 |
dtantsur | yoctozepto: ack, thank yoiu | 16:05 |
yoctozepto | mnasiadka: we have "all CI messy stuff" section | 16:06 |
yoctozepto | so it fits | 16:06 |
opendevreview | Radosław Piliszek proposed openstack/kolla master: Revert "Config: Set default home to /var/lib/{user}" https://review.opendev.org/c/openstack/kolla/+/805900 | 16:07 |
yoctozepto | mgoddard, mnasiadka: let's merge ^ | 16:08 |
yoctozepto | I blocked the related change to first test bifrost with that as well | 16:09 |
yoctozepto | the previous patches should be safe | 16:09 |
dtantsur | yoctozepto: configuring your IRC client to highlight known user names if very handy. there is a weechat extensions for that. | 16:09 |
yoctozepto | dtantsur: yeah, mine colours nicks but it chose a not-really-colourful colour for that folk :D | 16:10 |
yoctozepto | and I did not notice the query | 16:10 |
dtantsur | yep, can happen | 16:10 |
yoctozepto | so went "what the..." mode | 16:10 |
dtantsur | I've learned that anything can be a nick during my years here! :) | 16:11 |
yoctozepto | that true | 16:11 |
* dtantsur now stops distracting people from the pleasure of CI debugging | 16:11 | |
yoctozepto | :-) | 16:12 |
yoctozepto | revert proposed already, worry not | 16:12 |
mgoddard | cycle highlights: https://review.opendev.org/c/openstack/releases/+/806021 | 16:20 |
mgoddard | cycle lowlights: https://review.opendev.org/c/openstack/releases/+/8060212 | 16:21 |
opendevreview | Mark Goddard proposed openstack/kolla-ansible master: service-ks-register: ensure all user roles exist https://review.opendev.org/c/openstack/kolla-ansible/+/774451 | 16:23 |
opendevreview | Mark Goddard proposed openstack/kolla-ansible master: Support creating roles and assigning to service users https://review.opendev.org/c/openstack/kolla-ansible/+/774452 | 16:23 |
opendevreview | Mark Goddard proposed openstack/kolla-ansible master: docs: Add placeholder page for CI & testing information https://review.opendev.org/c/openstack/kolla-ansible/+/775564 | 16:25 |
opendevreview | Mark Goddard proposed openstack/kolla-ansible master: Performance: Split out prometheus node exporter and cadvisor https://review.opendev.org/c/openstack/kolla-ansible/+/740082 | 16:27 |
opendevreview | James Kirsch proposed openstack/kolla-ansible master: Support for keystone scoped authorization https://review.opendev.org/c/openstack/kolla-ansible/+/692179 | 16:36 |
mnasiadka | yoctozepto: messy +1 ;) | 17:00 |
opendevreview | Merged openstack/kolla master: Debian/Ubuntu: handle APT keys in proper way https://review.opendev.org/c/openstack/kolla/+/784923 | 17:18 |
yoctozepto | mgoddard: lowlights point nowhere ;-) | 17:21 |
mnasiadka | yoctozepto: that might be the goal of lowlights ;) | 17:24 |
yoctozepto | mnasiadka: I did not understand the joke and demand an immediate explanation! :D | 17:25 |
yoctozepto | mnasiadka: https://review.opendev.org/c/openstack/kolla/+/805900 | 17:42 |
yoctozepto | mnasiadka: let's merge that to rebuild tomorrow | 17:42 |
opendevreview | Radosław Piliszek proposed openstack/kolla-ansible master: [DNM] Test bifrost https://review.opendev.org/c/openstack/kolla-ansible/+/806040 | 18:22 |
opendevreview | Radosław Piliszek proposed openstack/kolla-ansible master: [DNM] Test bifrost https://review.opendev.org/c/openstack/kolla-ansible/+/806040 | 18:49 |
opendevreview | Merged openstack/kolla-ansible master: docs: Add placeholder page for CI & testing information https://review.opendev.org/c/openstack/kolla-ansible/+/775564 | 18:51 |
opendevreview | Merged openstack/kolla master: Revert "Config: Set default home to /var/lib/{user}" https://review.opendev.org/c/openstack/kolla/+/805900 | 20:34 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!