Wednesday, 2020-02-19

*** igordc has quit IRC00:06
*** k_mouza has joined #openstack-kolla00:23
openstackgerritJames Kirsch proposed openstack/kolla-ansible master: Add support for encrypting backend HAProxy traffic  https://review.opendev.org/66451600:27
*** k_mouza has quit IRC00:28
*** watersj has joined #openstack-kolla01:03
watersjhow do you reset /rebuild your rabbitmq cluster?01:11
*** dcapone2004 has joined #openstack-kolla01:32
*** skramaja has quit IRC01:51
*** JangwonLee_ has joined #openstack-kolla02:44
*** JangwonLee has quit IRC02:46
*** hongbin has joined #openstack-kolla02:53
cosmicsoundwatersj , you can always reconfigure kolla03:13
cosmicsoundthat will reconfigure your rabbitmq03:14
cosmicsoundor you could check with docker ps where it runs and remove it manually and redeploy03:14
*** hongbin has quit IRC03:23
openstackgerritYongjun Bai proposed openstack/kolla-ansible master: Add support for encrypting glance/heat api  https://review.opendev.org/70713104:04
*** shyamb has joined #openstack-kolla05:04
*** shyam89 has joined #openstack-kolla05:10
*** shyamb has quit IRC05:11
*** skramaja has joined #openstack-kolla05:25
*** evrardjp has quit IRC05:34
*** evrardjp has joined #openstack-kolla05:34
*** shyam89 has quit IRC06:04
*** dave-mccowan has joined #openstack-kolla06:05
*** shyamb has joined #openstack-kolla06:18
*** hongbin has joined #openstack-kolla06:23
*** hongbin has quit IRC06:28
*** jbadiapa has joined #openstack-kolla06:45
osmanlicilegimorning06:55
yoctozeptomorning06:55
osmanlicilegihow's going folks?06:56
osmanlicilegiif you ask me, I'll say I've missed kolla :]06:56
*** shyamb has quit IRC06:57
osmanlicilegifor the last few weeks I'm spending all of my time for the new datacenter drama06:57
*** shyamb has joined #openstack-kolla06:57
*** Tony31 has joined #openstack-kolla06:58
*** zhanglong has joined #openstack-kolla06:59
*** openstack has joined #openstack-kolla07:20
*** ChanServ sets mode: +o openstack07:20
mnasiadkaosmanlicilegi: so you’re not bored, right?:)07:22
*** openstack has joined #openstack-kolla07:44
*** ChanServ sets mode: +o openstack07:44
*** shyamb has quit IRC07:44
*** zzzeek has joined #openstack-kolla07:51
*** bengates has joined #openstack-kolla07:52
osmanlicilegisri_: if you compare kolla and tripleo, what would be your concerns about choosing tripleo over kolla.07:52
*** zzzeek has quit IRC07:54
*** zzzeek has joined #openstack-kolla07:56
sri_osmanlicilegi, I can't say much i am thinking about using tripleo just because of I've been using it for longer, there is lot of power failures in your datacenter is tripleo always comes up without any issues. I've never had change test k-a ansible on that level07:57
osmanlicilegigood point, power failues which I had several times for the last few months. I can clearly say the problematic part was often mariadb cluster and ceph.08:00
sri_"pcs status" "pcs resource cleanup" is lifesaver08:01
osmanlicilegiI was lucky k-a has mariadb_recovery and it always saved my life08:02
osmanlicilegialso, thanks yoctozepto for making mariadb deployment better than ever08:02
sri_osmanlicilegi, yes i've also used mariadb recovery in my lab it's always works08:02
sri_also lot of enterprise customers like some kind of technical support something like rhosp08:07
osmanlicilegiwe prefer spending the money for servers :]08:12
sri_:)08:15
osmanlicilegieither way, openstack is good. doesn't matter how you manage it...08:18
*** tonythomas has joined #openstack-kolla08:21
sri_true that08:21
*** zhanglong has quit IRC08:22
*** zhanglong has joined #openstack-kolla08:24
*** shyamb has joined #openstack-kolla08:26
*** rgogunskiy has joined #openstack-kolla08:30
*** rwellum_ has quit IRC08:34
*** mwhahaha has quit IRC08:34
*** alanmeadows has quit IRC08:34
*** rwellum_ has joined #openstack-kolla08:34
*** alanmeadows has joined #openstack-kolla08:35
*** egonzalez has quit IRC08:35
*** egonzalez has joined #openstack-kolla08:36
*** Tengu has quit IRC08:36
*** mwhahaha has joined #openstack-kolla08:37
*** rpittau|afk is now known as rpittau08:37
*** Tengu has joined #openstack-kolla08:38
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: CI: Add linuxbridge jobs  https://review.opendev.org/70831808:51
*** carli has joined #openstack-kolla08:54
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: CI: Remove dbus from bindep and playbooks  https://review.opendev.org/70808208:55
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Stop using deprecated stores and default_store in glance  https://review.opendev.org/70811408:56
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Allow setting additional galera WSREP options  https://review.opendev.org/70781709:00
*** dougsz has joined #openstack-kolla09:00
*** hjensas has joined #openstack-kolla09:09
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: CI: Add linuxbridge jobs  https://review.opendev.org/70831809:10
*** Jeffrey4l has quit IRC09:12
*** Jeffrey4l has joined #openstack-kolla09:13
*** Tony31 has quit IRC09:14
*** shyamb has quit IRC09:16
cosmicsoundgood day09:19
*** riuzen has joined #openstack-kolla09:25
*** gfidente has joined #openstack-kolla09:25
openstackgerritYongjun Bai proposed openstack/kolla-ansible master: Add support for encrypting glance/heat api  https://review.opendev.org/70713109:26
yoctozeptoosmanlicilegi: praise is always welcome!09:30
*** riuzen has quit IRC09:30
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: Deprecate deployment of MongoDB  https://review.opendev.org/70850809:30
*** AjiArya has joined #openstack-kolla09:36
*** k_mouza has joined #openstack-kolla09:37
*** Ajiarya7 has joined #openstack-kolla09:37
*** Ajiarya7 has left #openstack-kolla09:37
*** k_mouza has quit IRC09:37
*** k_mouza has joined #openstack-kolla09:38
AjiAryaHello guys, I'm having issue with deploying k8s cluster with magnum and octavia has anyone deployed it properly? I'm using kolla-ansible multinode09:40
AjiAryaHas anyone deployed magnum and octavia and runs properly?09:41
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Allow setting additional galera WSREP options  https://review.opendev.org/70781709:42
yoctozeptomnasiadka: any idea how https://review.opendev.org/#/c/623976/ was updated with no history entry?09:46
patchbotpatch 623976 - kolla-ansible - Convert '-' to '_' in ansible facts for interface ... - 6 patch sets09:46
yoctozeptoAjiArya: the fact is I don't remember anyone praising our magnum nor octavia deployment so it might be faily atm09:47
mnasiadkayoctozepto: better ask infra - I have no clue :)09:47
*** shyamb has joined #openstack-kolla09:50
AjiAryaOk Thanks for the kind answer, appreciate it.09:51
AjiAryaWhat is the channel of kolla-ansible infra, please?09:56
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Allow setting additional galera WSREP options  https://review.opendev.org/70781709:57
mnasiadkaAjiArya: that was not regarding your topic - you might want to share on paste.openstack.org what are your errors - if we won't be able to help you - we'll direct you to #openstack-containers (Magnum dev team channel)09:57
*** sri_ has quit IRC09:59
yoctozeptoAjiArya, mnasiadka: yes, all kolla questions are handled here :-)10:01
yoctozeptoAjiArya, mnasiadka: in the worst case we will ask you to file a bug on us10:01
*** shyamb has quit IRC10:03
AjiAryaThis is the error log http://paste.openstack.org/show/789740/10:05
yoctozeptoAjiArya: well, odd, I did not expect that :D which release is this? (kolla-ansible, openstack)10:08
yoctozeptoe.g. 9.0.1 + train10:08
yoctozeptoor something else10:08
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: CI: Add linuxbridge jobs  https://review.opendev.org/70831810:10
AjiAryaI'm using the latest version10:11
AjiAryakolla-ansible                9.0.110:11
AjiArya+ train10:11
mnasiadkaAjiArya: which os distro?10:13
AjiAryaCentOS10:13
*** zhanglong has quit IRC10:15
mnasiadkawell, then it's all clear10:16
AjiAryaWe've working on it for a little while but haven't figured out the error10:16
mnasiadka-rw-r--r-- 1 magnum magnum 2749 Feb 19 07:08 keystone_auth_default_policy.sample10:16
mnasiadkain the image10:16
mnasiadkaAjiArya: it's kolla-ansible bug - can please file one in bugs.launchpad.net/kolla-ansible ?10:16
AjiAryaIs there anyway I can fix it?10:18
AjiAryaOk Will be filed10:18
*** zhanglong has joined #openstack-kolla10:18
openstackgerritWill Szumski proposed openstack/kayobe master: Make docker registry volume configurable  https://review.opendev.org/70860710:19
mnasiadkaAjiArya: well, one this is that you're missing the /etc/magnum/keysdtone_auth_default_policy.json - this is not something you can fix easily10:19
yoctozeptomnasiadka: it's in etc this way?10:19
*** ivve has joined #openstack-kolla10:20
mnasiadkayoctozepto: yes, RDO delivers a bloody .sample10:20
yoctozeptomnasiadka: bah lol10:20
yoctozeptoAjiArya: CentOS binary or source?10:20
yoctozeptomnasiadka: what about source?10:20
mnasiadkayoctozepto: sorry, it's source - let me check binary10:20
yoctozeptomnasiadka: haha, then no blooding on rdo ;p10:21
AjiAryaI'm using source10:23
brtknrAjiArya: checkout https://github.com/RSE-Cambridge/cumulus-kayobe-config/blob/rocky/etc/kayobe/kolla/config/magnum/policy.json10:24
brtknrI am not sure if this is the solution10:24
mnasiadkayoctozepto: well, in binary even that file does not exist10:25
mnasiadkaso maybe Magnum is able to cope with it10:25
mnasiadkaseems the real issue is the 5th line in the paste - ssl certificate self signed10:25
yoctozeptobrtknr: well, random hints help from time to time anyway :-)10:26
yoctozeptomnasiadka: WARNING more important than ERROR? what world do we live in?10:27
yoctozeptomnasiadka: ah, it's just openstack, phew10:27
yoctozeptomnasiadka: that said, magnum bumps in my list of priority scenarios10:28
yoctozeptomnasiadka: after we handle the neutron bajzel10:28
mnasiadkayeah, I love bajzel10:28
mnasiadkaUssuri cycle is for getting neutron working properly I see :D10:29
AjiAryaI'll try brtknr10:29
brtknrAjiArya: sample keystone policy: https://seashells.io/v/FHVAknjH10:29
brtknrAjiArya: sample keystone policy: https://seashells.io/p/FHVAknjH10:30
brtknrthis is if you want to use k8s cluster using keystone credentials10:30
mnasiadkayoctozepto: you know that we also need to do octavia with magnum? :D10:30
yoctozeptomnasiadka: one step at a time, one step :-)10:31
yoctozeptomnasiadka: for getting neutron clean10:31
yoctozeptomnasiadka: we both already cleaned it up a lot10:31
yoctozeptomnasiadka: and it's getting ovn now10:31
yoctozeptomnasiadka: and we know we might be dropping vmware/hyperv there10:31
mnasiadkayeah, I need to remember to remove networking-ovn installation when we dump centos7/train on master :)10:31
yoctozeptomnasiadka: that said I don't remember if I acted on that thing to deprecate10:32
yoctozeptomnasiadka: why remove?10:32
yoctozeptowdym?10:32
mnasiadkayoctozepto: I think you can for sure deprecate hyperv - the company behind OpenStack work on Hyper-V didn't respond to my mail :)10:32
mnasiadkayoctozepto: networking-ovn has been moved to neutron base10:32
mnasiadkaas an in-tree driver10:33
mnasiadkabut since we use train in centos7 CI - it needs to stay there for a bit :)10:33
mnasiadkaI mean in our Dockerfile definitions10:33
yoctozeptomnasiadka: ah, this10:33
mnasiadkaalready on Ubuntu we have a bit of an issue, because we have the OVN driver in two places :)10:33
yoctozeptomnasiadka: well, osmanlicilegi said he wants to play with hyperv10:33
yoctozeptoor /me mistaken10:33
mnasiadkayoctozepto: he doesn't really need that hyperv role in k-a10:34
mnasiadkayoctozepto: and I doubt it even works10:34
mnasiadkaI mean configuring the Windows side of things using k-a10:34
*** shyamb has joined #openstack-kolla10:34
yoctozeptomnasiadka: I doubt it either10:34
mnasiadkaand it's even better to do it your own way, instead of relying on something totally not tested10:35
yoctozeptoosmanlicilegi: wdyt? ^ happy about deprec and removal for now?10:37
osmanlicilegiyoctozepto, mnasiadka: Alessandro Pilotti will join Open Infra Days Turkey and I will talk with him about supporting kolla. I will be clear then.10:37
mnasiadkaosmanlicilegi: so tell him to reply to my mail10:37
yoctozeptomnasiadka: you remember what was about vmware?10:37
osmanlicilegimnasiadka: I will :]10:37
yoctozeptomnasiadka: it's the same guy?10:37
osmanlicilegihttps://openinfradayturkey.com/speakers/alessandro-pilotti/10:38
mnasiadkayoctozepto: yeah, the guy I sent mail to - he's the CEO of CloudBase10:38
yoctozeptomnasiadka: CEOs are usually busy ;-)10:39
mnasiadkaosmanlicilegi: btw, you wanted someone from Kolla to run some session on that OpenInfra Day?10:39
osmanlicilegiI'm not sure if I can handle hyperv part alone with myself.10:39
osmanlicilegimnasiadka: tomorrow I'll have a meeting with the organizers and I will confirm what they want.10:39
*** k_mouza has quit IRC10:40
yoctozeptothey want mnasiadka for sure10:40
yoctozeptodead or alive :D10:40
yoctozeptoor not necessarily dead10:40
*** k_mouza has joined #openstack-kolla10:40
osmanlicilegiI know some sponsors will not join because of the wuhan crisis and this can result as limitting technical sessions.10:40
yoctozeptoI see10:41
osmanlicilegiIf they decide to limite technical sessions, I'm planning to organize a community meetup for kolla10:42
osmanlicilegiout of infra days of course...10:42
yoctozeptomnasiadka, osmanlicilegi: I replied to myself on ml10:45
yoctozeptomnasiadka, osmanlicilegi: will propose deprec some time later10:45
mnasiadkayoctozepto: replying to yourself is a bit funny, right?10:45
yoctozeptomnasiadka: well, there was no other reply to reply to :-)10:46
osmanlicilegiyoctozepto loves himself :]10:46
mnasiadkayoctozepto: feel free to deprecate, if there's no CI - it's really hard to claim that anything is working10:46
yoctozeptoosmanlicilegi: it's not illegal10:46
yoctozeptomnasiadka: yeah, I'm following cinder steps10:46
yoctozeptomnasiadka: one cannot expect to maintain this in blind10:47
*** pbing19 has quit IRC10:47
mnasiadkayoctozepto: yup10:48
openstackgerritMerged openstack/kolla-ansible master: Follow up on Ansible version checking  https://review.opendev.org/70517110:50
AjiAryaHere's the bug report https://bugs.launchpad.net/kolla-ansible/+bug/186387710:53
openstackLaunchpad bug 1863877 in kolla-ansible "Magnum SSL Error" [Undecided,New]10:53
openstackgerritMerged openstack/kolla master: change tripleo centos-7 build containers to centos-8  https://review.opendev.org/70821810:55
yoctozepto^ woop10:56
*** pbing19 has joined #openstack-kolla11:02
*** zhanglong has quit IRC11:02
openstackgerritMerged openstack/kolla stable/rocky: nova-libvirt: add UEFI packages to support UEFI instances  https://review.opendev.org/70606511:04
*** zhanglong has joined #openstack-kolla11:05
openstackgerritMerged openstack/kolla stable/train: CI: Replace cinder-lvm scenario with zun scenario  https://review.opendev.org/70656311:10
*** zhanglong has quit IRC11:13
mnasiadkaok, seems linuxbridge works11:15
mnasiadkaI'll add depends-on on the /run neutron change - let's see if it doesn't break anything11:15
*** zhanglong has joined #openstack-kolla11:15
mnasiadkabut still I need to create a bridge and veth pair in CI, so we could ssh into the vms11:15
yoctozeptomnasiadka: yeah, that's the next step11:21
yoctozeptomnasiadka: anything bad in the logs?11:21
yoctozepto2020-02-19 10:57:27.144 6 ERROR neutron.agent.linux.ip_lib [req-6900834a-6c89-4465-b174-b6f49db9d3a1 - - - - -] Device brqcd477d85-a2 cannot be used as it has no MAC address11:21
yoctozepto2020-02-19 10:57:40.357 6 ERROR neutron.agent.linux.ip_lib [req-6900834a-6c89-4465-b174-b6f49db9d3a1 - - - - -] Device brq15c017b1-74 cannot be used as it has no MAC address11:21
yoctozepto2020-02-19 10:57:24.796 6 ERROR neutron.agent.linux.ip_lib [req-8f9d9627-058e-42cf-a56a-6f161d23d1d0 - - - - -] Device ns-740b8018-95 cannot be used as it has no MAC address11:21
yoctozeptohmm, not promising xD11:21
yoctozeptolooks like linuxbridge on ubuntu creates same error messages as dhcp agent11:22
mnasiadkawell, I think those are some special ports, but why do they lack a mac address? that's at least weird11:22
yoctozepto"error" and "cannot be used" sound like "borky borky, fixy pleasy"11:23
yoctozeptothough it's not happening on centos11:23
yoctozeptolet's see there11:23
yoctozepto2020-02-19 11:15:00.637 8 ERROR neutron.agent.linux.ip_lib [req-a68aea40-6a2c-4597-b17a-19884d6694ae - - - - -] Device ns-3013809e-3c cannot be used as it has no MAC address11:25
yoctozepto2020-02-19 11:15:03.278 6 ERROR neutron.agent.linux.ip_lib [req-c49f623c-977e-4756-8c5d-e96fa29a4b81 - - - - -] Device brqdf3f666c-5b cannot be used as it has no MAC address11:25
yoctozepto2020-02-19 11:15:17.634 6 ERROR neutron.agent.linux.ip_lib [req-c49f623c-977e-4756-8c5d-e96fa29a4b81 - - - - -] Device brq2c514673-9b cannot be used as it has no MAC address11:25
yoctozepto2020-02-19 11:16:13.825 6 ERROR neutron.agent.linux.utils [req-c49f623c-977e-4756-8c5d-e96fa29a4b81 - - - - -] Exit code: 255; Stdin: ; Stdout: ; Stderr: Unknown argument: '--among-src'.11:25
yoctozepto2020-02-19 11:21:11.911 6 ERROR nova.compute.manager [req-8755f595-53e0-40cc-9ef2-108075b2f6a2 86833bf449414ff0971b753c0c361be8 c415f21b52a14e6bbb7ef1814b0c17a4 - default default] [instance: b4557d6a-8fe5-4468-92ef-85f6ae79a30d] Instance failed to spawn: nova.exception.VirtualInterfaceCreateException: Virtual Interface creation failed11:25
yoctozeptomnasiadka: not really worky ^11:25
mnasiadkaunknown argument - I love it11:25
yoctozeptoand somehow it managed to squeeze out dhcp agent error on it, very nice11:25
yoctozeptonotice it did not happen with ovs11:26
yoctozeptodid it silently ignore us due to wrong mtu?11:26
yoctozeptomnasiadka: it could be that c8 changed bridging utils enough to break it11:26
yoctozeptostill, the dhcp case is very interesting11:27
mnasiadkaubuntu succeeded, centos8 failed, let's see if centos7 also fails11:28
*** hjensas has quit IRC11:30
yoctozeptomnasiadka: indeed11:31
yoctozeptomnasiadka: I replied to your last comment, not agreed / confused11:32
yoctozeptomnasiadka: passed11:33
mnasiadkaso centos8 is special11:33
yoctozeptomnasiadka: yeah11:33
yoctozeptomnasiadka: ubuntu breaks a little11:34
yoctozeptomnasiadka: and c8 completely11:34
yoctozeptomnasiadka: answer my comment and let's move on to vxlan refactoring and more thorough connectivity testing11:34
*** AjiArya has quit IRC11:37
*** rlljorge has joined #openstack-kolla11:39
*** rpittau is now known as rpittau|bbl11:40
rlljorgeHello there, I have multiples backends in Cinder, I created 2 types for set one as a default I put default_volume_type=Standard in cinder-volume.conf but not work, this is the correct way ?11:42
yoctozeptorlljorge: it does not work in what sense of not working?11:42
yoctozeptoerror message, data loss, no effect, nude fairies in terminal?11:43
rlljorgeyoctozepto When I create a new volume without specify the volume-type the volume did not have a volume-type11:45
yoctozeptorlljorge: as in horizon? it does not read that file so it has no idea11:47
yoctozeptorlljorge: ah, I see you set it in cinder-volume, I believe cinder wants it set in api11:49
yoctozeptoas it's api that decides11:49
rlljorgeyoctozepto http://paste.openstack.org/show/789744/11:50
yoctozeptorlljorge: yeah, I reflected above11:51
yoctozeptorlljorge: configure cinder-api with this param11:51
yoctozeptoand then reconfigure11:51
rlljorgeyoctozepto ok ! I will try now ! Thanks11:52
paddyoniellquestion about the development workflow for kolla-ansible, during installation I ran pip install inside the kolla-ansible repo directory, which installs the playbooks to /usr/local/share/kolla-ansible. If I make changes to ansible playbooks in the git repo do I need to rerun pip install to update the playbooks in /usr/local/share/kolla-ansible?11:55
yoctozeptopaddyoniell: yeah, if you want to act on it this way11:55
openstackgerritYongjun Bai proposed openstack/kolla-ansible master: Add support for encrypting glance/heat api  https://review.opendev.org/70713111:55
yoctozeptopaddyoniell: it's more efficient to do an editable install (-e) and run from tools/kolla-ansible11:56
yoctozeptothen you get both current code and playbooks :-)11:56
paddyoniellyoctozepto: thanks, will try the editable install, didn't know that was a thing.11:57
*** shyamb has quit IRC11:57
openstackgerritKevin Zhao proposed openstack/kolla-ansible stable/rocky: Load known, standard kernel modules from the host, not within containers  https://review.opendev.org/70862811:59
*** hjensas has joined #openstack-kolla12:03
*** shyamb has joined #openstack-kolla12:07
watersjis there any persistent data with in rabbitmq cluster? meaning if I destroyed the containers and redeployed rabbit cluster will start up properly again12:09
*** carli has quit IRC12:10
*** shyamb has quit IRC12:14
*** mgoddard has quit IRC12:16
*** kplant has joined #openstack-kolla12:16
osmanlicilegiwatersj: it will work. maybe you will need to restart some services after the new rabbitmq deployment.12:17
osmanlicilegiwatersj: the keypoint is just removing the containers is not enough for a fresh rabbitmq deployment. you also need to remove its volumes.12:17
osmanlicilegiwatersj: docker stop rabbitmq && docker rm rabbitmq && docker volume rm rabbitmq12:18
watersjosmanlicilegi, thx. I had to repurpose one of my controllers and it magically was the master for rabbitmq. Of course i discovered this after the fact12:20
watersjthen power outage12:20
*** mgoddard has joined #openstack-kolla12:21
*** skramaja has quit IRC12:30
*** zhanglong has quit IRC12:32
mnasiadkayoctozepto: the ebtables missing --amon-src seems to have a corresponding bug in redhat - https://bugzilla.redhat.com/show_bug.cgi?id=172063712:33
openstackbugzilla.redhat.com bug 1720637 in nftables "Ebtables don't have support for the among match" [Medium,Closed: duplicate] - Assigned to psutter12:33
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: CI: Add linuxbridge jobs  https://review.opendev.org/70831812:45
*** zhanglong has joined #openstack-kolla12:48
*** ktibi has joined #openstack-kolla12:52
openstackgerritMerged openstack/kolla-ansible master: Allow setting additional galera WSREP options  https://review.opendev.org/70781712:56
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: CI: Add linuxbridge jobs  https://review.opendev.org/70831812:57
mnasiadkamgoddard: I won't be available for the meeting later tonight, I haven't done anything with the centos8 backports to train, just rechecked some of them, will work on it today/tomorrow12:58
yoctozeptomgoddard: but mnasiadka did well on neutron stuff12:58
yoctozeptowatersj: there is no "master" concept in rabbitmq12:59
yoctozeptoall are created equal12:59
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: CI: Add linuxbridge jobs  https://review.opendev.org/70831813:01
*** rpittau|bbl is now known as rpittau13:02
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: CI: Fine tune Galera gmcast.peer_timeout to 15 seconds  https://review.opendev.org/70781813:05
*** pbing19 has quit IRC13:10
yoctozeptomnasiadka: how is tripleo handling this regarding linuxbridge compat?13:13
yoctozeptomnasiadka: could you test debian as well?13:17
mnasiadkaI can do anything :)13:18
yoctozeptomnasiadka: cool, lemme send you my todo and you can start right away13:19
mnasiadkayoctozepto: yeah, right ;)13:20
*** kplant has quit IRC13:20
rlljorgefolks, there any documetaion in kolla for complete setup for octavia ?13:21
*** kplant has joined #openstack-kolla13:23
*** AjiArya has joined #openstack-kolla13:28
yoctozeptorlljorge: nope :-)13:29
mgoddardhi all13:33
rlljorgeyoctozepto deploy octavia is a journey ...13:34
yoctozeptorlljorge: openstack is a journey, we write this poem every day :-)13:34
yoctozeptomgoddard: well, hello13:35
mgoddardyoctozepto: \o13:35
yoctozeptoo/13:35
mgoddardlooks like it's been busy13:35
mgoddardI won't read scrollback as I don't have all day, but is there anything worth sharing?13:35
yoctozeptomgoddard: ppl praising our work13:41
mgoddardI thank them for their praise on behalf of all kolla contributors13:41
yoctozeptomgoddard: with mnasiadka we are on better testing of networking13:41
mgoddard+1 for that13:42
*** AjiArya has quit IRC13:42
*** AjiArya has joined #openstack-kolla13:44
*** AjiArya has quit IRC13:45
*** jlfont_ has joined #openstack-kolla13:50
*** jlfont_ is now known as jlfont13:51
jlfontHi! I have been tinkering recently with kolla-ansible and Monasca (stein). I am wondering if it is possible to just enable monasca-agent for an kolla-ansible controlled cluster, using an external stand-alone Monasca instance13:54
mgoddarddougsz: ^13:54
jlfontI have seen that enabling monasca enables the whole stack of services13:54
mgoddardjlfont: dougsz will confirm, but I don't think that's possible currently. Seems like a reasonable request though13:55
jlfontmgoddard, thanks :-)13:56
yoctozeptomgoddard, mnasiadka: https://bugs.launchpad.net/kolla-ansible/+bug/186390113:57
openstackLaunchpad bug 1863901 in kolla-ansible ussuri "[Ussuri] Ceph recommendations break nova direct snapshot" [Medium,Triaged]13:57
yoctozeptothat's what one gets following best practices13:57
yoctozeptobEsT pRaCtIcEs13:57
jlfontdoing this with logging is kind of easy, cause fluentd is already there taking care of the logs, you only have to change its configuration to point to whatever monasca13:58
dougszjlfont: Enabling just the monasca-agent in a Kolla deployment isn't currently supported, although you might be able to get it to work by removing the other Monasca services from the inventory. I expect what you really want is a flag, enable_monasca_agent.13:59
jlfontdougsz: yes, I was wondering if something like that was contemplated13:59
jlfontI will try as you suggest14:00
dougszIt seems like a reasonable thing to support to me.14:00
dougszWhat we typically do is use an ansible role outside of Kolla to deploy it14:00
dougszWe use this: https://github.com/stackhpc/ansible-role-monasca-agent14:01
dougszI expect you want logs as well?14:01
dougszThe Monasca Fluentd output config depends on the enable_monasca flag, but that could be changed, or you could use custom Fluentd config.14:01
jlfontdougsz: yes, initially I have been just hacking the fluentd conf for the logs14:02
jlfontwith good results14:02
dougszGood to hear :)14:02
jlfontwe already have a OpenStack cloud running, and we want to create a separated Monasca cluster, just enabling monasca-agent, like we can do with fluent would be super convenient14:03
*** rlljorge has quit IRC14:04
dougszYeah - it makes sense, for one to get Monasca off the control plane HAProxy14:04
*** rlljorge has joined #openstack-kolla14:04
*** rlljorge has left #openstack-kolla14:05
jlfontthanks a lot guys, I will report how it went :-)14:05
dougszPlease do, thanks jlfont14:06
*** rlljorge has joined #openstack-kolla14:09
*** oneswig has joined #openstack-kolla14:11
yoctozeptomnasiadka: and it passed (linuxbridge) - so removal of that mount is good to go14:13
*** dcapone2004 has quit IRC14:13
*** rlljorge has quit IRC14:14
yoctozeptomnasiadka: well, not quite, needs misc cleaning before approval14:14
yoctozeptomnasiadka: I have one more thing to check there as well14:14
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: WIP: Remove useless neutron-linuxbridge-agent env vars  https://review.opendev.org/70865814:19
yoctozeptomnasiadka: ^14:19
*** zhanglong has quit IRC14:22
*** zhanglong has joined #openstack-kolla14:25
*** hjensas has quit IRC14:31
openstackgerritMerged openstack/kayobe master: Use latest release of StackHPC iDRAC role  https://review.opendev.org/70834714:41
*** zhanglong has quit IRC14:41
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Improve Ansible version check error handling  https://review.opendev.org/70526114:47
*** TrevorV has joined #openstack-kolla14:48
mgoddardmgoddard mnasiadka hrw egonzalez yoctozepto rafaelweingartne cosmicsound osmanlicilegi14:54
mgoddardmeeting in 5 ^14:54
mnasiadkayoctozepto: makes sense14:55
*** irclogbot_3 has quit IRC14:56
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: CI: Add linuxbridge jobs  https://review.opendev.org/70831814:57
mgoddard#startmeeting kolla15:00
openstackMeeting started Wed Feb 19 15:00:16 2020 UTC and is due to finish in 60 minutes.  The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot.15:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:00
*** openstack changes topic to " (Meeting topic: kolla)"15:00
openstackThe meeting name has been set to 'kolla'15:00
mgoddard#topic rollcall15:00
*** openstack changes topic to "rollcall (Meeting topic: kolla)"15:00
mgoddard\o15:00
cosmicsoundo/15:00
osmanlicilegio/15:00
dougsz\o15:00
osmanlicilegio/15:01
yoctozeptoo/15:01
*** irclogbot_1 has joined #openstack-kolla15:02
mgoddard#topic announcements15:02
*** openstack changes topic to "announcements (Meeting topic: kolla)"15:02
mgoddard#undo15:02
openstackRemoving item from minutes: #topic announcements15:02
mgoddard#topic agenda15:02
*** openstack changes topic to "agenda (Meeting topic: kolla)"15:02
mgoddard* Roll-call15:02
mgoddard* Announcements15:02
mgoddard* Review action items from last meeting15:02
mgoddard* CI status15:02
mgoddard* Ussuri release planning (kolla & kolla ansible)15:02
mgoddard* Ussuri release planning (kayobe)15:02
mgoddard** XenAPI deprecated in Nova Train, removal pending15:02
mgoddard* Kolla SIG (aka Kolla Klub?) https://etherpad.openstack.org/p/kolla-sig15:02
mgoddard#topic announcements15:02
*** openstack changes topic to "announcements (Meeting topic: kolla)"15:02
mgoddardLooks like we have none on the agenda. Anyone have any?15:03
osmanlicileginope15:03
dougszI was going to raise an issue, but I think I have a workaround.15:04
dougszso, nope, for now at least15:05
mgoddardWe can discuss in open discussion if you like15:05
mgoddard#topic review action items from last meeting15:05
*** openstack changes topic to "review action items from last meeting (Meeting topic: kolla)"15:05
mgoddardyoctozepto to deprecate mongodb for ubuntu, drop for centos15:05
mgoddardI think I saw it proposed.15:05
mgoddardgood work15:06
mgoddard#topic CI status15:06
*** openstack changes topic to "CI status (Meeting topic: kolla)"15:06
mgoddard"general issues with new py, affects tripleo as well, and jobs used from k-a"15:06
mgoddardwhat is this?15:06
mgoddardstill an issue?15:06
mgoddardanyone?15:08
osmanlicilegiI'm not sure about the details15:08
mgoddardkolla CI looks green15:08
*** dave-mccowan has quit IRC15:08
mgoddarddate is 13/2/2020, possibly an old one15:09
yoctozeptowe got past that15:09
mgoddardcool15:09
mnasiadkayeah, but still the workaround is applied I guess15:09
mnasiadkawith installing newer virtualenv or something15:09
yoctozeptoit is15:09
yoctozeptotesting revert now15:09
mgoddardoh it's that thing15:09
yoctozeptolooks passy15:09
yoctozeptohttps://review.opendev.org/#/c/707623/15:10
patchbotpatch 707623 - kolla - Revert "Upgrade virtualenv in pre" - 1 patch set15:10
mnasiadkayoctozepto: kolla-ansible part as well? :)15:10
yoctozeptowell, you won't see from here unless you have zuul plugin in browser15:10
yoctozeptoyeah15:10
mgoddardok, let's try to get those merged15:10
mgoddardonto k-a15:10
mgoddardstein ubuntu binary fails on neutron migrations (foreign key blah blah)15:10
yoctozeptoit failed recently yeah15:11
mgoddardI remember seeing it15:11
mgoddardis it a persistent failure?15:11
yoctozeptolet's check :-)15:11
yoctozeptohttps://zuul.opendev.org/t/openstack/builds?job_name=kolla-ansible-ubuntu-binary&branch=stable%2Fstein15:11
mgoddardlooks bad15:12
mgoddardanyone want to investigate?15:12
mnasiadkaneutron backported the fail to stein? :D15:12
yoctozeptorechecking https://review.opendev.org/#/c/706078/15:12
patchbotpatch 706078 - kolla-ansible (stable/stein) - Fix multiple issues with MariaDB handling - 1 patch set15:12
yoctozeptomnasiadka: no idea15:12
yoctozeptomnasiadka: somehow only binary15:13
mgoddardI wonder why it's binary only. Maybe the package needs a bump?15:13
mnasiadkaI already raised two bugs to neutron, not gonna look into third one :)15:13
mgoddardor stein needs a release15:13
yoctozeptoin uca15:13
mgoddardyes15:13
yoctozeptobut they surely should know theu have b0rken debs15:13
yoctozeptolet's se15:13
mnasiadkawe could check the neutron version in UCA stein-testing - if there's a version difference15:14
mgoddardsounds like a volunteer15:14
mgoddard#action mnasiadka to check ubuntu stein UCA neutron versions and request release if necessary15:15
mgoddardthanks :)15:15
mgoddardreport back if it's more complicated than that15:15
openstackgerritYongjun Bai proposed openstack/kolla-ansible master: Add support for encrypting glance/heat api  https://review.opendev.org/70713115:15
mgoddard"general issues with new py when run from kolla side"15:15
openstackgerritYongjun Bai proposed openstack/kolla-ansible master: Add support for encrypting glance/heat api  https://review.opendev.org/70713115:16
mgoddardassume this is the virtualenv issue?15:16
yoctozeptomgoddard: the same15:16
mgoddardremoving15:16
mnasiadkamgoddard: 14.04 got to updates some time ago, probably it's broken-ish15:16
mgoddard14.04??15:16
mnasiadkaneutron-server 14.0415:16
mnasiadkawhatever it is on Ubuntu side :)15:17
mgoddardlol, thought you meant ubuntu 14.0415:17
mnasiadkahttp://reqorts.qa.ubuntu.com/reports/ubuntu-server/cloud-archive/stein_versions.html15:17
yoctozeptomnasiadka: and what is current15:17
mgoddard14.1.015:18
mnasiadkaI don't like the 1 in the middle :)15:18
yoctozepto14.0.4 vs 14.1.015:18
yoctozeptomnasiadka: why?15:18
yoctozeptoit's a bumpy in reqs most likely15:18
mnasiadkait basically means normally it's a big change or some feature backport :)15:18
yoctozeptoor exactly fix for our issue which required dep update ;-)15:19
*** shyamb has joined #openstack-kolla15:19
yoctozeptoanyways, mnasiadka pokes uca and we are good15:19
mnasiadkamight be15:19
mgoddard#undo15:19
openstackRemoving item from minutes: #link http://reqorts.qa.ubuntu.com/reports/ubuntu-server/cloud-archive/stein_versions.html15:19
mnasiadkawell, another bug to raise I guess15:19
mgoddard#undo15:19
openstackRemoving item from minutes: #action mnasiadka to check ubuntu stein UCA neutron versions and request release if necessary15:19
mgoddard#link http://reqorts.qa.ubuntu.com/reports/ubuntu-server/cloud-archive/stein_versions.html15:19
mgoddard#action mnasiadka request neutron 14.1.0 in stein UCA15:19
yoctozeptopretty precise15:20
mgoddardprecise pangolin?15:20
mgoddardanyways15:20
yoctozeptoxD15:20
mgoddard#topic Ussuri release planning (kolla & kolla ansible)15:20
*** openstack changes topic to "Ussuri release planning (kolla & kolla ansible) (Meeting topic: kolla)"15:20
mgoddardAny Ussuri features we need to discuss this week?15:21
mnasiadkamaybe not a feature, but a community goal - can I get some reviews on https://review.opendev.org/#/c/707800/ ?15:21
patchbotpatch 707800 - kolla - [community goal]: Add contributor and PTL guide - 2 patch sets15:21
mgoddardwill try to get a looko15:22
mnasiadkagood15:22
yoctozepto"So You Want to Contribute..." is in the goal? ;D15:22
mnasiadkawould love some reviews on OVN patch15:22
mgoddardmy review queue seems to be growing faster than I can process15:22
mnasiadkayoctozepto: complain to the goal coordinator :)15:22
mgoddardmnasiadka: I started a review of OVN :)15:23
mnasiadkamgoddard: good to hear :)15:23
osmanlicilegiah I feel ashame now. mnasiadka I'm so sorry I couldn't spent time for the comments you wait from me :|15:23
mgoddardwill try to finish15:23
mnasiadkaosmanlicilegi: easy, fix your datacenter :)15:24
mgoddardcan we mark https://blueprints.launchpad.net/kolla-ansible/+spec/ceph-ansible complete?15:24
mnasiadkawell, I'd like to add prechecks for keys existence - but it's rather optional and nice to have15:24
mgoddardand https://blueprints.launchpad.net/kolla-ansible/+spec/remove-ceph15:24
mnasiadkayeah, that one for sure15:25
mgoddardmnasiadka: the blueprint is just about CI15:25
mnasiadkamgoddard: right, I just took the extra mile to make it more user friendly :)15:25
mgoddardalways appreciated15:25
mnasiadkawhoa, remove-ceph was never approved :)15:25
yoctozeptomnasiadka: WHAT HAVE YOU DONE?!15:26
mnasiadkayoctozepto: well, it's better to beg for forgiveness, than to ask for permission - that's my life motto :)15:26
yoctozeptomnasiadka: gets things done faster15:27
mgoddardrm -rf; echo sorry15:27
mnasiadkayeah, sorry is the key15:27
yoctozeptomgoddard: I've almost spilled my drink15:27
mgoddardapologies15:27
mgoddardalthough I did already say sorry15:28
mgoddardanything else to discuss?15:29
yoctozeptoonly stuff for later15:29
mgoddard#topic Ussuri release planning (kayobe)15:29
*** openstack changes topic to "Ussuri release planning (kayobe) (Meeting topic: kolla)"15:29
mgoddarddougsz15:30
mgoddardCan't see priteau or jovial15:30
yoctozeptowhere them folks at15:30
yoctozeptowhere, where15:30
dougszhmm.. what do we have, possibly a cells config cleanup15:31
*** priteau has joined #openstack-kolla15:32
mgoddarddougsz: what needs to be done for that?15:32
cosmicsoundSo ceph stays? :D15:32
*** jovial[m] has joined #openstack-kolla15:32
* cosmicsound chuckels15:32
mgoddardcosmicsound: ceph? too late - it's gone15:32
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: CI: Fine tune Galera gmcast.peer_timeout to 15 seconds  https://review.opendev.org/70781815:32
mgoddardaha, priteau & jovial[m]15:32
priteauo/15:32
jovial[m]hi15:33
dougszmgoddard: Anything to make it more manageable for high numbers of cells15:33
dougszon the topic of ceph.. do we have any Kayobe references to remove?15:33
mgoddardthere is more work cells planned for k-a, will you be looking at it before ussuri?15:34
dougszWe need shared cell controllers sooner or later15:34
dougszI will try to15:34
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: CI: Add linuxbridge jobs  https://review.opendev.org/70831815:34
dougszWe already have some half backed patches around that15:35
mgoddardyep15:35
dougsz*baked15:35
mgoddardI think a separate nova-compute role might make it more scalable, but that needs investigation15:35
dougszagreed15:35
mgoddardon the ceph removal, I created https://storyboard.openstack.org/#!/story/200729515:35
jovial[m]a high priority for me would be to implement the custom extension points proposal15:36
mgoddardwe have a role to label block devices15:36
mgoddardanyone want to pick up ceph removal?15:36
mgoddardshould be therapeutic if you like removing code15:36
jovial[m]I will volunteer if no one else wants it15:36
mgoddard#action jovial[m] to remove kayobe ceph block device labelling support https://storyboard.openstack.org/#!/story/200729515:37
mgoddardthanks15:37
dougszthanks jovial[m]15:37
yoctozeptomgoddard: is it possible to volunteer from outside?15:37
yoctozeptoah, too late :-)15:37
mgoddardanyone can volunteer to work on kayobe15:37
mgoddardthere is no outside in open source :)15:37
yoctozeptowell, you know, making it official15:37
yoctozeptoand stuff15:37
yoctozeptocould get to know kayobe better15:38
mgoddardyou want to take it?15:38
mgoddardI'm sure jovial[m] won't mind too much :)15:38
yoctozeptomgoddard: yeah, therapeutic15:38
mgoddard#undo15:38
openstackRemoving item from minutes: #action jovial[m] to remove kayobe ceph block device labelling support https://storyboard.openstack.org/#!/story/200729515:39
jovial[m]fine by me :)15:39
mgoddard#action yoctozepto to remove kayobe ceph block device labelling support https://storyboard.openstack.org/#!/story/200729515:39
mgoddardthanks yoctozepto15:39
dougsz+115:39
yoctozepto;-)15:39
mgoddardjovial[m] is away for a while anyway15:39
mgoddardjovial[m]: you would like to see the custom extension points proposal15:40
*** Trevor_V has joined #openstack-kolla15:40
mgoddardhttps://storyboard.openstack.org/#!/story/200166315:40
cosmicsound:)15:40
dougszWe certainly have an issue with exporters, would be nice to use it for that15:40
mgoddardjovial[m]: want to pick it up?15:40
jovial[m]for sure -  lots of people want this I believe15:41
mgoddard+115:41
*** Trevor__V has joined #openstack-kolla15:41
mgoddardit will make for a smoother ride in many cases15:41
priteauExtension points and multiple environments are two big requested features15:42
mgoddard#action jovial[m] to work on custom extension points15:42
mgoddardyep15:42
yoctozeptofolks, remember some extensions might better fit in k-a15:42
yoctozeptoand these are welcome as well15:42
yoctozeptomgoddard to correct me :-)15:42
mgoddardyoctozepto: you are right15:42
mgoddardwe will try to use judgement15:42
yoctozeptoOK15:42
mgoddardthis is about a framework for running hook playbooks at various points in kayobe workflows15:43
*** TrevorV has quit IRC15:43
yoctozeptomgoddard: yeah, skimmed over story15:43
mgoddardonto the other big feature then - multiple environments15:44
mgoddardhttps://storyboard.openstack.org/#!/story/200200915:44
mgoddardanyone interested in picking this up?15:44
mgoddardit's a bigger task15:44
dougsznot enough cycles, sorry15:44
mgoddardcould potentially be a collaboration between multiple people15:44
*** Trevor_V has quit IRC15:44
dougszMaybe someone is willing to help fund it?15:45
mgoddardI'd like to be involved to some extent, but CentOS 8 is eating my cycles at the moment15:45
jovial[m]another must have feature in my opinion15:46
mgoddardI think we have a funder (getting a bit internal here though :))15:46
*** shyamb has quit IRC15:46
mgoddardok, no takers yet15:46
mgoddardkeep it in mind15:46
yoctozeptomgoddard: slightly off topic; bored with bifrost, how is bikolla going? I really liked the "containerized bifrost"15:46
jovial[m]how long have we got until the next release?15:46
mgoddardKolla feature freeze: Apr 27 - May 0115:47
yoctozeptomgoddard: has it any roadmap or whatever?15:47
mgoddardKolla RC1: May 11 - May 1515:47
mgoddardKolla final release: Jun 1 - Jun 515:47
yoctozeptocan we fit it any place?15:47
mgoddardwhen you say containerised bifrost you don't mean our current bifrost container :)15:47
yoctozeptomgoddard: I mean bikolla hence ""15:47
yoctozepto:D15:47
mgoddarddougsz was looking at it recently15:48
yoctozeptodougsz: how's bikolla? :-)15:48
mgoddardstill at the proposal/design stage though IIRC15:48
mgoddardor possibly just blocked on cycles :)15:48
yoctozeptoI'd like to be in for any eventual discussion15:48
dougszwe need it for deploying a large number of hypervisors15:48
dougszI was also wondering about just using the top level ironic for that15:49
mgoddardnoted yoctozepto15:49
yoctozeptothanks15:49
yoctozeptodougsz: what do you mean by "top level"15:49
dougszthe overcloud ironic, not bifrost ironic15:50
mgoddardthat is the other option15:50
mnasiadkadougsz: don't you want to use it for standing up the overcloud? :)15:50
yoctozeptoyeah, I'm confused too15:50
dougszwell, bifrost can still handle a small number of controllers, just not the hypervisors15:51
yoctozeptoI thought bikolla is about deploying standalone ironic with kolla15:51
yoctozeptoso bifrost via kolla without bifrost15:51
mgoddardcorrect15:51
mnasiadkadougsz: looking from Kolla side - I would be happy to deprecate and remove bifrost container, and this way we would need to maintain it :)15:51
yoctozepto+115:52
dougszyeah, it is balancing the maintenance of bifrost against the work required for bikolla15:52
jovial[m]would kayobe need a name change? ;-)15:52
yoctozeptojovial[m]: not it's a snake15:52
yoctozeptobe become bikolla15:52
mgoddardkayobikolla?15:53
yoctozeptokolla on bi...kolla on bi..kolla .....15:53
yoctozeptoperpetum kayobile15:53
jovial[m]:D15:53
mnasiadkamgoddard: looking from another perspective - maybe it would make sense to document bikolla in k-a and make it easier to use for users? I don't think anybody is really aware of using bikolla instead of bifrost.15:53
yoctozepto^ that's also what I meant15:53
yoctozeptomnasiadka better at putting my thoughts in words15:53
yoctozeptothanks mnasiadka15:53
mgoddardwe could do that15:54
yoctozepto"but we won't"15:54
mnasiadkawell, anybody with spare cycles can do that15:54
yoctozeptomnasiadka to pick it up?15:54
yoctozeptoI am too nooby atm ;p15:54
mnasiadkaI can add it to my list, although not with a very high priority :)15:54
yoctozeptohigh is fine ;-)15:55
mnasiadkahell, we could even do deploy-bikolla with some stock inventory file and stock globals to deploy it on localhost15:55
mgoddarddocs to deploy the containers should be easy enough15:55
mgoddardusing the thing and a full tested procedure would take longer15:56
yoctozeptothat said, does kayobe have docs on how ci is configured?15:56
yoctozeptoaka what is tested15:56
yoctozepto(exactly what k-a misses ;-) )15:56
yoctozeptoand how15:56
mgoddardit's covered as well as in k-a :)15:56
yoctozeptoack ;-)15:57
mgoddardlet's move on15:57
mgoddard#topic XenAPI deprecated in Nova Train, removal pending15:57
yoctozepto3 minutes15:57
*** openstack changes topic to "XenAPI deprecated in Nova Train, removal pending (Meeting topic: kolla)"15:57
yoctozeptoindeed let's15:57
yoctozeptoquickie from me15:57
yoctozeptowe seem to use xenapi with our xen integration15:57
yoctozeptoI want you to confirm that15:57
yoctozepto(or deny)15:57
mgoddardlooks like it15:57
yoctozeptoand then we deprecate and remove following nova15:57
yoctozeptothank you15:57
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Stop using deprecated stores and default_store in glance  https://review.opendev.org/70811415:57
*** rgogunskiy has quit IRC15:58
mgoddardcertainly lots of greps for xenapi in k-a15:58
yoctozeptowell, that was fast15:58
yoctozeptomnasiadka: your thoughts?15:58
yoctozeptojovial[m], priteau, dougsz, osmanlicilegi: any kolla xen experience?15:58
mgoddardare they removing completely, or extracting into a separate lib?15:58
dougsznope15:58
yoctozeptomgoddard: they deprecated for removal in future15:58
mnasiadkait was not really clear what are they doing15:58
osmanlicilegiI'm not using xen since 2015 :]15:58
priteaunone, sorry15:58
jovial[m]afraid not, I'ma  libvirt only man15:58
yoctozeptoin *Train*15:58
mnasiadkawe don't have xen CI15:58
mnasiadkathere was somebody that was trying to make it work15:59
egonzalezcberent (non online now) i believe he o some of his clients were using xen15:59
yoctozeptomnasiadka: thanks, sherlock!15:59
mgoddardyeah, JamesBenson maybe?15:59
mnasiadkamgoddard: that was the guy - not overly active now15:59
yoctozeptowonder how they doing15:59
mgoddardhi egonzalez :)15:59
yoctozeptoegonzalez: what is full name of cberent?15:59
mgoddardmaybe we should wait for nova removal?15:59
yoctozepto(or was)15:59
dougszIs there time for a quick question about /etc/security/limits.conf?15:59
mnasiadkamgoddard: I would just deprecate for now16:00
egonzalezyoctozepto: Christian Berendt16:00
mnasiadkawith some nice warning message when enabled16:00
mgoddardyoctozepto: berendt is his IRC nick16:00
mgoddarddeprecation makes sense16:00
yoctozeptoegonzalez, mgoddard: oh, that's the guy who started helping with reviews again16:00
yoctozeptomnasiadka: ^16:00
mgoddard(which is what yoctozepto said)16:00
mgoddardyes16:00
mnasiadkayeah, Christian got a bit active today16:00
mgoddard#topic open discussion16:01
*** openstack changes topic to "open discussion (Meeting topic: kolla)"16:01
mgoddarddougsz has a qq16:01
yoctozeptonice reviews, we should keep him in core team as we can't add him more16:01
dougsz1. deployment uses base image with some settings in /etc/security/limits.conf16:01
generalfuzzIf we could have a couple eyes on https://review.opendev.org/664516 it would be greatly appreciated16:01
patchbotpatch 664516 - kolla-ansible - Add support for encrypting backend HAProxy traffic - 19 patch sets16:01
yoctozeptoas for xen - I'll post it to ml16:01
dougsz2. Settings make their way into nova-ssh container16:02
mnasiadka(they make their way to all container I guess)16:02
dougsz3. One of the settings is nproc, the number of processes a user can open16:02
dougsz4. That setting restricts the number of SSH sessions16:02
dougszSo.. should the Nova SSH container own that file, to prevent this?16:02
yoctozeptogeneralfuzz: we can't handle that root thingy with current sudo?16:03
dougszShould we support customising it explicity?16:03
dougszOr just use custom config in the Docker file to modify it in the Nova SSH container?16:03
yoctozeptogeneralfuzz: I mean, it breaks current assumptions we use sudo to gain root in containers16:03
mnasiadkadougsz: What is their setting of max concurrent live migrations?16:03
cosmicsounddougsz , +1 that could be also a security issue16:03
mgoddarddougsz: I think the problem is that VMs and other processes are created as the nova user, which is also used by the nova-ssh container16:03
dougszyeah16:04
mnasiadkamgoddard: I don't know if VMs and other processes use a workflow to get pam_limits in the way, or not16:04
mnasiadkamgoddard: without using dimensions it's a tricky thing16:04
yoctozeptorebuild is pain for this, let's make it customizable too16:04
mnasiadkawell, another thing - why not use docker dimensions a.k.a. docker run --ulimit for that?16:05
generalfuzzyoctozepto: I will investigate it with sudo16:06
mgoddardmnasiadka: I imagine that would be a different cgroup-based limit16:06
yoctozeptogeneralfuzz: thanks, that would limit the scope of the patch, otherwise we need splitting AGAIN :D16:06
mnasiadkamgoddard: are you sure? https://docs.docker.com/engine/reference/commandline/run/#set-ulimits-in-container---ulimit16:06
mnasiadkawell, dimensions are other thing - I confused everybody I think :)16:07
*** iniazi has quit IRC16:07
mnasiadkamgoddard: this way instead of copying file, we could set Ulimits via Docker API16:07
mgoddardmnasiadka: dimensions == ulimit16:08
mgoddardthat would be nicer16:08
mgoddardand also easy to verify16:08
openstackgerritWill Szumski proposed openstack/kolla-ansible master: Introduce influxdb_datadir_volume  https://review.opendev.org/70786116:08
mnasiadkamgoddard: well, I thought dimensions == --cpu-percent and others from Docker CLI - then I confused myself even more :)16:08
yoctozeptooh my16:09
yoctozeptoso much confusion16:09
yoctozepto:S16:09
mgoddardhttps://docs.openstack.org/kolla-ansible/rocky/reference/resource-constraints.html16:09
mnasiadkaI would ask why it's called dimensions instead of ulimits :)16:09
mnasiadkamgoddard: so then I'm right16:09
mgoddarddoesnhttps://docs.openstack.org/kolla-ansible/latest/reference/deployment-config/resource-constraints.html16:09
mgoddardhttps://docs.openstack.org/kolla-ansible/latest/reference/deployment-config/resource-constraints.html16:09
mnasiadkathese are not ulimits, just cgroups resource constraints16:09
mnasiadkaso I was thinking correct16:10
mgoddardhas ulimits16:10
mgoddardadded in stein?16:10
mnasiadkathen it's an easy thing16:10
mnasiadkajust use ulimits in the dimensions16:10
mgoddardyes. question is whether it should be default or local config16:10
mnasiadkainstead of doing it with pam_limits16:10
mnasiadkamgoddard: well then, it's up to the user/operator - they can use a default/global and only override locally on nova_ssh (or nova_*)16:11
mgoddardyeah, although it was quite a nasty bug to track down - maybe we should fix it for everyone? or provide a tuneable?16:12
mnasiadkacare for an example? The only thing I can think of is warning if there are constrains defined in pam_limit :)16:12
*** TrevorV has joined #openstack-kolla16:13
yoctozeptoso outside docker ulimit overrides limits from internal security?16:13
yoctozepto(I lost track of discussion due to all confusion)16:13
mnasiadkayoctozepto: well now it's a good question, which one wins16:14
mnasiadka:)16:14
yoctozeptoah-ha16:14
mnasiadkaBe careful setting nproc with the ulimit flag as nproc is designed by Linux to set the maximum number of processes available to a user, not to a container. For example, start four containers with daemon user.16:14
mnasiadkathis is also a good note from the documentation16:14
mnasiadkamaybe we should add it to our docs as well16:14
mnasiadkaand recommend using Docker API based ulimits, instead of delivering that file16:15
mnasiadkathat in a file16:15
mgoddardthis is why VM procs in nova_libvirt affect nova_ssh16:15
*** Trevor__V has quit IRC16:15
mgoddardI think some investigation needs to be done into how to fix this16:16
yoctozeptouse a separate user?16:16
mnasiadkahow? :)16:16
mgoddardsame user - nova16:16
mgoddardalthough I would exect UID namespaces to differ16:16
mgoddardseems PAM doesn't care :)16:16
yoctozeptomnasiadka: I'm adding confusion, don't confuse me16:16
yoctozeptohmm, odd16:17
mnasiadkamgoddard: well, it's PAM...16:17
mgoddardI think there are too many unknowns to discuss this now16:17
yoctozeptoneeds investigation and let's end the meeting16:17
mgoddardsomeone needs to investigate and report back16:17
mgoddardare you picking that up dougsz?16:17
yoctozeptodougsz?16:17
mnasiadkayes, dougsz came with a question, so he should investigate :D16:17
dougszI will start by writing a bug report :)16:17
yoctozeptomnasiadka: I wrote that at the same time ;-)16:18
yoctozeptomnasiadka: it's asking dougsz ;-)16:18
dougszand do some investigation16:18
yoctozeptodougsz: bug report +116:18
mgoddard#action dougsz to write bug report about nova SSH nproc issue16:18
yoctozeptootherwise no problem16:18
yoctozepto:-)16:18
dougszthanks guys16:18
yoctozeptothanks dougsz16:18
mgoddard18 minutes over, what chatterboxes16:19
mgoddardthanks all16:19
mgoddard#endmeeting16:19
*** openstack changes topic to "Kolla IRC meetings on Wednesdays @ 15:00 UTC - see agenda @ https://goo.gl/OXB0DL | Whiteboard: https://etherpad.openstack.org/p/KollaWhiteBoard | IRC channel is *LOGGED* @ http://goo.gl/3mzZ7b"16:19
openstackMeeting ended Wed Feb 19 16:19:12 2020 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:19
openstackMinutes:        http://eavesdrop.openstack.org/meetings/kolla/2020/kolla.2020-02-19-15.00.html16:19
mnasiadkathanks16:19
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/kolla/2020/kolla.2020-02-19-15.00.txt16:19
openstackLog:            http://eavesdrop.openstack.org/meetings/kolla/2020/kolla.2020-02-19-15.00.log.html16:19
mgoddardOne day I will forget to end the meeting and we will have a week long meeting16:19
osmanlicilegi:)))16:20
yoctozeptoshappens ;-)16:20
osmanlicilegi"the never ending meeting of kolla"16:20
osmanlicilegithx and good evening folks16:22
cosmicsoundneed to prepare my designate findings and tls / ssl issues16:23
cosmicsoundopened a bug yet surely can get more details for it16:23
mnasiadkamgoddard: I think we have week long meetings already, just we have an hour (which we exhaust) of ,,official meeting'' :)16:24
*** TrevorV has quit IRC16:27
*** bengates has quit IRC16:32
*** TrevorV has joined #openstack-kolla16:33
cosmicsoundquestion, after your experience what is most reasonable storage backend for openstack?16:37
cosmicsoundso much wanted to give a try to ceph, yet more and more people told me is not best compared to price and size.16:38
egonzalezoverall ceph, but Im happy with netapp in prod now16:38
cosmicsoundnetapp paid solution or FOSS software16:40
cosmicsoundAww appliances16:40
egonzalezpaid $$$$, nfs version16:41
*** ivve has quit IRC16:41
yoctozeptocosmicsound: what's wrong with ceph anyways?16:42
cosmicsoundyoctozepto , from what i've researched around it, its much more complex on planning and more costly it seems16:45
cosmicsoundanyhow will surely give it a try together with k-a and ceph-ansible16:45
cosmicsoundfor the moment is still working well16:45
*** mcnugit has joined #openstack-kolla16:46
*** rpittau is now known as rpittau|afk16:48
cosmicsoundegonzalez , this are all SAN based?16:49
openstackgerritMark Goddard proposed openstack/kolla-ansible master: WIP: Try waiting for Monasca API to come up  https://review.opendev.org/65652816:53
openstackgerritMark Goddard proposed openstack/kolla-ansible master: WIP: CI: Add monasca scenario  https://review.opendev.org/64989316:53
*** igordc has joined #openstack-kolla16:59
*** jlfont has quit IRC17:08
cosmicsoundegonzalez , thanks for feedback, surely will check all options before picking any17:13
yoctozeptomgoddard, mnasiadka, dougsz: btw, k-a centos8 cells jobs are really faily, not sure why17:15
mgoddardhmm17:15
mnasiadkaHuh, novncconsole leak17:16
mnasiadkaI mean novncproxy17:17
*** cah_link has quit IRC17:20
*** gfidente has quit IRC17:24
*** tonythomas has quit IRC17:26
*** TrevorV has quit IRC17:33
*** evrardjp has quit IRC17:34
*** evrardjp has joined #openstack-kolla17:34
openstackgerritJason Anderson proposed openstack/kolla-ansible master: Update docs to say that external Ceph required  https://review.opendev.org/70870817:35
yoctozeptomnasiadka: as long as you don't let randoms read your logs you are fine17:38
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Clean up some kolla-kubernetes cruft in OVS  https://review.opendev.org/70871017:39
yoctozeptomgoddard joins in cleanup competition :-)17:40
* mgoddard never stopped cleaning up :)17:41
yoctozeptomnasiadka: re ceph keyrings - does not ansible decrypt on copy?17:41
yoctozeptomnasiadka: so they could be just encrypted?17:42
mnasiadkaThey could, putting them in passwords.yml won’t help with anything - still you need to template them out to files on computes:)17:43
mnasiadkayoctozepto: I’ve done so much with ceph recently, that I’m happy to see what others would want to do :)17:44
yoctozeptomnasiadka: yeah, but I think his problem is not existent :-)17:45
openstackgerritMark Goddard proposed openstack/kolla master: Remove start-ovsdb-server script from openvswitch-db-server  https://review.opendev.org/70871117:45
*** igordc has quit IRC17:46
*** k_mouza_ has joined #openstack-kolla17:46
yoctozeptooh no, efried is leaving us :<17:47
*** k_mouza has quit IRC17:49
*** dougsz has quit IRC17:51
*** k_mouza_ has quit IRC17:51
*** rlljorge has joined #openstack-kolla17:58
*** TrevorV has joined #openstack-kolla18:01
openstackgerritMerged openstack/kolla master: Get rid of Python 2 support  https://review.opendev.org/69131618:01
*** oneswig has quit IRC18:03
*** paddyoniell is now known as paddyoneill|afk18:14
*** ktibi has quit IRC18:20
*** igordc has joined #openstack-kolla18:22
*** priteau has quit IRC18:27
*** priteau has joined #openstack-kolla18:28
*** jbadiapa has quit IRC18:32
*** ivve has joined #openstack-kolla18:32
*** priteau has quit IRC18:33
*** mcnugit has quit IRC18:44
*** k_mouza has joined #openstack-kolla18:47
*** mlukas has joined #openstack-kolla18:48
*** k_mouza has quit IRC18:50
*** rgogunskiy has joined #openstack-kolla19:05
*** mlukas has quit IRC19:08
*** rgogunskiy has quit IRC19:09
*** hjensas has joined #openstack-kolla19:13
openstackgerritRadosław Piliszek proposed openstack/kolla master: Drop weird swift rootwrap config (xenapi with placeholder values)  https://review.opendev.org/70873319:54
yoctozeptomgoddard, mnasiadka: '"networking-hyperv>=5.0.0,<6.0.0"' <- that's on kolla master, pretty unsupported :D20:07
yoctozeptoI'm proposing deprecs but hyperv looks like immediate drop to me ;p20:08
yoctozeptoC:\Program Files\Cloudbase Solutions\OpenStack\Nova\bin\mkisofs.exe20:11
yoctozeptopretty interesting20:11
kplantew, hyperv20:16
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: Deprecate Hyper-V integrations  https://review.opendev.org/70874020:18
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: Deprecate VMware integrations  https://review.opendev.org/70874120:20
bjolois kolla-ansible planning to support OVN?20:26
mnasiadkayoctozepto: mkisofs for config drive, and yes - this constraint of version is weird20:26
mnasiadkabjolo: yes, there is a change under review, so it should land in Ussuri20:26
yoctozeptoah, mnasiadka was faster :-)20:27
bjolook cool :)20:27
bjololooking forward to test it out20:27
noxoid_https://review.opendev.org/#/c/696841/20:27
patchbotpatch 696841 - kolla-ansible - OVN Support - 92 patch sets20:27
noxoid_for the curious20:27
yoctozeptoit's mnasiadka's work btw, so he gets most praise :-)20:27
*** rlljorge has quit IRC20:29
noxoid_indeed, the ovn documentation shows a lot of benefits compared to a pure neutron SDN. i plan to look at migrating to ovn once ussuri has been out for a few months20:30
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: Deprecate XenAPI integrations  https://review.opendev.org/70874320:33
yoctozeptomnasiadka: all deprecated20:34
yoctozeptomnasiadka: we might want to write up more details about these but I think it's a decent start20:35
yoctozeptomnasiadka: especially when nobody cares ;-)20:35
bjolokudos to mnasiadka :)20:35
yoctozeptokudos indeed20:35
bjoloovs is a pain. we have a lot of weird behaviors20:36
yoctozeptobjolo: well ovn is ovs in a nicer packaging20:38
yoctozeptomnasiadka: INFO:kolla.common.utils.opendaylight:[91mhttps://nexus.opendaylight.org/content/repositories/opendaylight-carbon-epel-7-x86_64-devel/org/opendaylight/integration-packaging/opendaylight/6.5.0-0.1.20180430snap36.el7.noarch/opendaylight-6.5.0-0.1.20180430snap36.el7.noarch.rpm: [Errno 12] Timeout on20:38
yoctozeptohttps://nexus.opendaylight.org/content/repositories/opendaylight-carbon-epel-7-x86_64-devel/org/opendaylight/integration-packaging/opendaylight/6.5.0-0.1.20180430snap36.el7.noarch/opendaylight-6.5.0-0.1.20180430snap36.el7.noarch.rpm: (28, 'Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds')20:38
yoctozeptonice way to end the day :-)20:38
*** kplant has quit IRC20:41
*** k_mouza has joined #openstack-kolla20:51
*** primeministerp has joined #openstack-kolla20:52
*** primeministerp has quit IRC20:53
*** k_mouza has quit IRC20:56
*** emccormick has joined #openstack-kolla21:09
openstackgerritJason Anderson proposed openstack/kolla-ansible master: [gnocchi] Disable statsd daemon by default  https://review.opendev.org/67161821:13
openstackgerritJason Anderson proposed openstack/kolla-ansible master: [gnocchi] Disable statsd daemon by default  https://review.opendev.org/67161821:29
*** ne0h|nihilist has joined #openstack-kolla21:54
openstackgerritChristian Berendt proposed openstack/kolla master: Add a generic header  https://review.opendev.org/70875821:59
*** TrevorV has quit IRC22:21
ne0h|nihilisthi, is there a way to verify if a custom configuration I'm making is being merged or applied to the container? Im trying to configure an external iscsi storage backend in cinder.22:28
*** ne0h|nihilist has quit IRC22:43
*** rgogunskiy has joined #openstack-kolla22:45
*** rgogunskiy has quit IRC22:49
*** diurnalist has joined #openstack-kolla23:11
*** diurnalist has quit IRC23:14
*** k_mouza has joined #openstack-kolla23:20
*** k_mouza has quit IRC23:24
*** AjiArya has joined #openstack-kolla23:44
*** diurnalist has joined #openstack-kolla23:49

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!