Wednesday, 2020-02-26

*** lile has quit IRC00:10
*** cah_link has quit IRC00:13
*** diurnalist has quit IRC00:35
*** diurnalist has joined #openstack-kolla00:37
*** sean-k-mooney has joined #openstack-kolla00:39
*** k_mouza has joined #openstack-kolla00:45
*** k_mouza has quit IRC00:49
*** sean-k-mooney has quit IRC00:59
*** diurnalist has quit IRC01:11
*** diurnalist has joined #openstack-kolla01:18
*** diurnalist has quit IRC01:49
*** mrunge has quit IRC02:30
*** mrunge has joined #openstack-kolla02:32
*** zhanglong has joined #openstack-kolla03:13
*** zhanglong has quit IRC03:36
*** zhanglong has joined #openstack-kolla03:37
openstackgerritHongbin Lu proposed openstack/kolla master: Zun: add zun-cni-daemon image  https://review.opendev.org/70827303:48
*** hongbin has joined #openstack-kolla03:49
openstackgerritHongbin Lu proposed openstack/kolla-ansible master: [WIP] Zun: Add zun-cni-daemon to compute node  https://review.opendev.org/70821304:15
*** factor has quit IRC04:31
*** factor has joined #openstack-kolla04:31
openstackgerritYongjun Bai proposed openstack/kolla-ansible master: WIP:Add support for encrypting glance/heat api  https://review.opendev.org/70713104:32
openstackgerritHongbin Lu proposed openstack/kolla-ansible master: [WIP] Zun: Add zun-cni-daemon to compute node  https://review.opendev.org/70821304:34
*** hongbin has quit IRC04:39
*** zhanglong has quit IRC04:53
*** zhanglong has joined #openstack-kolla04:55
openstackgerritJames Kirsch proposed openstack/kolla-ansible master: Add support for encrypting backend HAProxy traffic  https://review.opendev.org/66451605:02
*** evrardjp has quit IRC05:34
*** evrardjp has joined #openstack-kolla05:35
*** skramaja has joined #openstack-kolla05:35
*** zhanglong has quit IRC05:58
*** zhanglong has joined #openstack-kolla06:01
*** sri_ has joined #openstack-kolla06:04
*** kozhukalov has joined #openstack-kolla06:08
*** zhanglong has quit IRC06:20
*** zhanglong has joined #openstack-kolla06:22
*** shyamb has joined #openstack-kolla06:28
*** tony31 has joined #openstack-kolla06:37
*** k_mouza has joined #openstack-kolla06:46
*** k_mouza has quit IRC06:51
*** cah_link has joined #openstack-kolla07:09
*** shyamb has quit IRC07:14
yoctozeptomorning07:32
sri_morning07:40
sri_yoctozepto, sorry for silly question, how many controller we required to run 30 to 50 compute nodes with DVR enabled07:45
yoctozeptosri_: even one controller might be just fine, not sure if dvr changes anything in that regard; for ha you might still want more, probably 3 for default service placement07:49
sri_yoctozepto, planing to use 3 controller for sure, my workload is very generic, create a bunch of vm and just running schedule snapshots and backups.07:52
sri_yoctozepto, with dvr the network traffic don't need to the controllers07:53
cosmicsoundgood day07:54
*** dciabrin has joined #openstack-kolla07:54
sri_I think 3 controller should be ok as you mentioned07:55
yoctozeptosri_: it never has to go there, you probably mixed controller with networking node07:55
sri_yoctozepto, yes network with mixed with controller nodes07:56
cosmicsoundyoctozepto , how can we debug better this designate deploy process in kolla, there are some weird issues here to look into, last time i had in designate_worker another pool id from the one in passwords.yml. now the pool id its good and still say no pool available07:59
*** bengates has joined #openstack-kolla08:02
tony31morning08:03
yoctozeptocosmicsound: you can run services with debug=True if logs are not satisfactorily detailed08:07
*** tonythomas has joined #openstack-kolla08:11
*** shyamb has joined #openstack-kolla08:21
cosmicsoundI have the enable_debug true08:22
cosmicsoundisnt this same as you mentioned?08:22
cosmicsoundyoctozepto , i found a issues08:24
cosmicsoundIn designate-sink default.conf we have this08:24
cosmicsoundhttps://mdb.uhlhost.net/uploads/fce149d037300410/image.png08:24
cosmicsoundif you check [handler:nova_fixed] / zone_id and [handler:neutron_floatingip] / zone id08:24
cosmicsoundThey are not filled in08:24
cosmicsoundNow how do I debug this why is not filled, could it be a error in playbook?08:25
cosmicsoundsame issues we debated last time with yankcrime08:25
mnasiadkacosmicsound: do you see this? https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/designate/templates/designate.conf.j2#L6608:34
mnasiadkawe have no automation to fill in the zone-id, you need to do it by yourself in /etc/kolla/config/designate.conf08:34
cosmicsoundits missing only 2 locations in fact in designate-sink08:36
cosmicsoundthe rest of the values are filled in08:37
cosmicsoundit helped me last time by adding the values in designate-sink.conf08:37
cosmicsoundand reconfigure testing it now as we speak08:37
mnasiadkacosmicsound: as I said, kolla-ansible will leave it blank, you need to configure it somewhere in /etc/kolla/config08:42
cosmicsoundright08:51
cosmicsoundediting the template also helps08:51
cosmicsoundor il try08:51
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Fix fernet bootstrap and key distribution - follow up  https://review.opendev.org/70708008:52
*** shyamb has quit IRC08:55
mgoddardmorning09:01
tony31morning09:06
*** lennyb has quit IRC09:06
*** lennyb has joined #openstack-kolla09:07
tony31when I run "kayobe overcloud host configure" it tries to run some variables as commands once it finishes. I only noticed this start today - I think I borked the control host 🙈09:07
tony31`(kayobe) [cv-user@juc-kach1-prd kayobe]$ hilosipip_interface: bond2bash: hilosipip_interface:: command not found`09:08
yoctozeptobond2bash :D09:08
tony31:)09:08
yoctozeptoBond, Bash Bond09:09
yoctozepto;-)09:09
tony31usually it's bin bash09:09
tony31🗑️09:09
well100Morning09:15
*** dougsz has joined #openstack-kolla09:15
openstackgerritMark Goddard proposed openstack/kolla stable/train: CentOS 8: base and openstack-base images  https://review.opendev.org/70953709:17
openstackgerritMark Goddard proposed openstack/kolla stable/train: CentOS 8: Update packages in images  https://review.opendev.org/70920209:17
well100mgoddard i have a doing to my patch?09:19
*** ktibi has joined #openstack-kolla09:20
yoctozeptowell100: yeah, there is review/testing progress09:22
well100yoctozepto: okay fine im wait for instruciton :-)09:28
openstackgerritMark Goddard proposed openstack/kolla-ansible stable/train: Python 3: Use distro_python_version for WSGI python_path  https://review.opendev.org/70971309:34
hrwwell100: https://docs.openstack.org/tripleo-docs/latest/contributor/contributions.html09:39
hrwops09:40
hrwwell100: I meant https://wiki.openstack.org/wiki/How_To_Contribute09:40
openstackgerritMark Goddard proposed openstack/kolla-ansible master: CI: CentOS 8: Enable TLS on core scenario  https://review.opendev.org/70999409:44
mgoddardwell100: hi. Are you asking what needs to be done to your patch?09:45
openstackgerritMark Goddard proposed openstack/kolla-ansible master: DNM: Testing cloudkitty in master  https://review.opendev.org/68155509:46
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Fix fernet bootstrap and key distribution - follow up  https://review.opendev.org/70708009:46
openstackgerritMark Goddard proposed openstack/kolla-ansible master: DNM: Testing cloudkitty in master  https://review.opendev.org/68155509:47
yoctozeptohrw: sending folks to tripleo? HOW DARE YOU :D09:49
mgoddardwell100: yoctozepto has posted some comments on your patch09:50
mgoddard(assume we are talking about https://review.opendev.org/#/c/707379)09:50
patchbotpatch 707379 - kolla-ansible - Cloudkitty cant not conncet to Auth - 2 patch sets09:50
hrwcan't not connect?09:53
well100mgoddard: should i make a new patch without v3?10:04
openstackgerritAlfredo Moralejo proposed openstack/kolla master: Use StorageSIG repos for Ceph in CentOS8  https://review.opendev.org/70733810:06
openstackgerritMichal Nasiadka proposed openstack/kayobe master: Allow setting pip_proxy  https://review.opendev.org/70958010:08
openstackgerritMichal Nasiadka proposed openstack/kayobe master: Allow setting pip_proxy  https://review.opendev.org/70958010:10
tony31hi kayobe guys10:11
tony31I managed to trace back this problem which I am having at the moment. I am using one physical node for 2 roles: compute and storage. I found that the tunnel network is not being set on the compute node any longer. Not sure how long this issue has been there. But if I remove the storage role from this node then it successfully gets the tunnel10:12
tony31network10:12
openstackgerritMark Goddard proposed openstack/kolla master: CentOS 8: Actually disable EPEL, and epel-modular  https://review.opendev.org/71000310:12
mgoddardhi tony31, I see. It is normally best to only have one top level role per node10:13
mgoddarddue to the way we build our lists of network interfaces etc10:13
tony31fair enough :)10:13
mgoddardI'd suggest using compute, and adding storage bits as necessary10:13
tony31trying to make the most of the hardware I have - I'll try and spin up a storage VM on another hypervisor10:14
tony31thanks for the advice on it10:14
mgoddardtony31: you could add the compute group to kolla_overcloud_inventory_storage_groups10:15
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Fix fernet bootstrap and key distribution - follow up  https://review.opendev.org/70708010:15
tony31hmm10:15
mgoddardand add networks to compute_extra_network_interfaces as necessary10:15
tony31is that only in the venv ?10:16
tony31`$ grep -rni -e "kolla_overcloud_inventory_storage_groups"venvs/kayobe/share/kayobe/ansible/group_vars/all/kolla:275:kolla_overcloud_inventory_storage_groups:venvs/kayobe/share/kayobe/ansible/group_vars/all/kolla:298:      "{{ kolla_overcloud_inventory_storage_groups }}"`10:16
mgoddardtony31: we don't include it in etc/kayobe/kolla.yml but it seems to be required quite frequently so maybe we should add it10:17
mgoddardtony31: but in general you can add arbitrary variables to the files in etc/kayobe/*.yml10:17
mgoddardwe just include the most common ones10:17
tony31Would it work the other way around? if I add `storage_extra_network_interfaces:` and list the tunnel network there?10:18
*** hjensas has quit IRC10:18
*** k_mouza has joined #openstack-kolla10:21
*** hjensas has joined #openstack-kolla10:21
*** k_mouza has quit IRC10:21
*** k_mouza has joined #openstack-kolla10:21
openstackgerritMark Goddard proposed openstack/kolla-ansible master: DNM: Test swift s3api  https://review.opendev.org/70973010:24
tony31mgoddard - I added `storage_extra_network_interfaces:` and set the tunnel interface there (in a host_vars) and now when I do a host configure, it is including the tunnel network. This is just for testing at the moment. I'm still pulling the pieces together.10:27
tony31i just needed a storage node in the same area that the storage is located so that the build is fast - iscsi over a VPN tunnel was a bit slow (for glance building instances)10:28
tony31:)10:28
openstackgerritMark Goddard proposed openstack/kolla master: CentOS 8: Actually disable EPEL, and epel-modular  https://review.opendev.org/71000310:29
*** skramaja has quit IRC10:30
openstackgerritMark Goddard proposed openstack/kolla-ansible stable/train: CI: Use upper constraints when installing clients  https://review.opendev.org/70975110:34
*** tony31 has quit IRC10:35
hrwbbl10:37
openstackgerrityatin proposed openstack/kolla master: Build collectd image for CentOS8  https://review.opendev.org/70970110:41
openstackgerrityatin proposed openstack/kolla master: Build collectd image for CentOS8  https://review.opendev.org/70970110:41
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: OVN Support  https://review.opendev.org/69684110:41
openstackgerrityatin proposed openstack/kolla master: Build collectd image for CentOS8  https://review.opendev.org/70970110:42
*** sean-k-mooney has joined #openstack-kolla10:45
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Fix fernet bootstrap and key distribution - follow up  https://review.opendev.org/70708010:52
openstackgerritMerged openstack/kolla stable/train: vitrage-base: honor distro_python_version  https://review.opendev.org/70955310:57
openstackgerritMerged openstack/kolla-ansible master: Fix RabbitMQ hostname address resolution precheck  https://review.opendev.org/70789210:57
*** priteau has joined #openstack-kolla11:01
well100i have question to rdns :-)11:09
well100how do i create the dns zone for it to work11:10
openstackgerritMerged openstack/kolla-ansible stable/train: Allow to override external network params in init-runonce  https://review.opendev.org/70967111:21
openstackgerritMerged openstack/kolla-ansible master: CI: Fix TLS upgrade test  https://review.opendev.org/70980811:22
*** shyamb has joined #openstack-kolla11:27
*** shyamb has quit IRC11:45
openstackgerritMark Goddard proposed openstack/kolla stable/train: CentOS 8: Remove shellinabox from ironic-conductor  https://review.opendev.org/70920311:59
openstackgerritMark Goddard proposed openstack/kolla stable/train: CentOS 8: Use upstream Ceph/master  https://review.opendev.org/70920411:59
openstackgerritMark Goddard proposed openstack/kolla stable/train: CentOS 8: Allow SSH access to keystone_ssh and nova_ssh  https://review.opendev.org/70920511:59
openstackgerritMark Goddard proposed openstack/kolla stable/train: CentOS 8: Relax ironic iPXE EFI bootloader location  https://review.opendev.org/70920711:59
openstackgerritMark Goddard proposed openstack/kolla stable/train: Switch to python3 in bindep.txt  https://review.opendev.org/70975711:59
openstackgerritMark Goddard proposed openstack/kolla stable/train: CentOS 8: Add deploy jobs in CI  https://review.opendev.org/70953811:59
mgoddardyoctozepto, mnasiadka, osmanlicilegi: https://review.opendev.org/#/q/topic:bp/centos-rhel-8+status:open+branch:stable/train12:00
*** shyamb has joined #openstack-kolla12:08
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: OVN Support  https://review.opendev.org/69684112:22
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: CI: Add addressing on external network  https://review.opendev.org/70936212:27
openstackgerritMerged openstack/kolla-ansible stable/train: Python 3: Use distro_python_version for monasca agent CA file  https://review.opendev.org/70971112:31
openstackgerritMerged openstack/kolla-ansible stable/train: Remove unused python path calculation from vmtp  https://review.opendev.org/70971212:31
openstackgerritMerged openstack/kolla-ansible stable/train: Use local python interpreter for keystone cron generator  https://review.opendev.org/70971412:31
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Fix fernet bootstrap and key distribution - follow up  https://review.opendev.org/70708012:34
openstackgerritMerged openstack/kolla-ansible stable/train: Support python 3 in kolla-ansible script  https://review.opendev.org/70971612:36
ktibiHello guys, I add a param in kayobe for the registry password. But I can see in the password.yml of kolla, the password is set to null. How can I fix that ? Need to edit manualy the password.yml ?12:36
ktibior maybe I can regenerate the file :/12:39
mgoddardktibi: I'd suggest using this: https://docs.openstack.org/kayobe/latest/configuration/kolla-ansible.html#configuring-custom-passwords12:43
mgoddardmaybe we need to add that to the defaults?12:43
ktibiok strange because I have set docker_registry_password (I can see it in global of kolla) but when kayobe check if passwords.yml is correct, kayobe override and add None to the password12:44
ktibimgoddard, ok because kayobe use "{{ temp_path }}/passwords.yml" and I think I need to modify this file :p12:46
ktibiok found, need ot modify etc/kayobe/kolla/password ;)12:49
*** kplant has joined #openstack-kolla12:57
openstackgerritMark Goddard proposed openstack/kolla master: Actually disable EPEL, and epel-modular  https://review.opendev.org/71000312:58
openstackgerritMark Goddard proposed openstack/kolla master: Throw TypeError in repository enable/disable macros  https://review.opendev.org/71002712:58
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: OVN Support  https://review.opendev.org/69684112:59
openstackgerritMichal Nasiadka proposed openstack/kayobe master: Add framework to deploy user-defined containers on seed  https://review.opendev.org/70968913:01
openstackgerritPiotr Rabiega proposed openstack/kolla master: Add collectd-pcie-errors package for PCIe Errors (C7 only)  https://review.opendev.org/71002913:06
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Fix fernet bootstrap and key distribution - follow up  https://review.opendev.org/70708013:06
well100ptr reverse dns can you help me?13:07
cosmicsoundwell100 , what is wrong13:10
cosmicsoundi work also now on designate13:10
well100do I have to create the ptr zone, if so where do I have to create it?13:11
cosmicsoundyou can do it in /etc/hosts locally or with you provider, not sure how your domain is routed13:13
cosmicsoundi use for example a dns in cpanel now, that i forward to my server where i make the reverse dns13:13
cosmicsoundand then i add it locally in /etc/hosts13:13
cosmicsoundto reflect the ip of the main bind9 host in designate13:14
cosmicsoundin my case that is dns.uhlhost.net if you do a host dns.uhlhost.net you see it points to my designate ip for this.13:14
cosmicsoundnot sure if this helps13:14
cosmicsoundalso if you have more ips your provider should allow you to change the reverse records, once ofcourse a dns points to them13:15
cosmicsoundor maybe using the bind templates13:15
cosmicsoundyou could change there all ptr records13:15
*** skramaja has joined #openstack-kolla13:16
well100no13:26
well100I want designate to create the PTR records of fixed and floating ips13:27
openstackgerritMerged openstack/kolla-ansible stable/train: Python 3: Use distro_python_version for WSGI python_path  https://review.opendev.org/70971313:35
osmanlicilegimnasiadka: after merging https://review.opendev.org/#/c/707375/ I've discovered ubuntu cannot manage network namespaces anymore. should be same on debian too.13:38
patchbotpatch 707375 - kolla-ansible - Change /run bind mount for neutron/openvswitch (MERGED) - 4 patch sets13:38
osmanlicilegiseems /run rules ubuntu13:38
osmanlicilegiI'll raise a bug report soon13:39
yoctozeptoosmanlicilegi: dang13:41
yoctozeptoosmanlicilegi: that's odd because it is passing in CI13:42
yoctozeptowonder which part broke there13:42
openstackgerritYongjun Bai proposed openstack/kolla-ansible master: WIP:support for separate admin vip address and admin fqdn  https://review.opendev.org/71003613:42
*** diurnalist has joined #openstack-kolla13:44
openstackgerritMerged openstack/kolla stable/train: Remove tgtd and scsi-target-utils support in CentOS/RHEL 8  https://review.opendev.org/70242113:48
*** diurnalist has quit IRC13:48
mgoddard:( ubuntu deploy jobs seem broken on master13:50
*** shyamb has quit IRC13:51
yoctozeptowhere? how?13:53
mgoddardpossible it's caused by dropping epel :)13:54
mgoddardit's probably fine13:54
mgoddardhold up that doesn't make sense on ubuntu!13:55
mgoddardwake up mgoddard13:55
openstackgerritMark Goddard proposed openstack/kolla master: Actually disable EPEL, and epel-modular  https://review.opendev.org/71000313:59
openstackgerritMark Goddard proposed openstack/kolla master: Throw TypeError in repository enable/disable macros  https://review.opendev.org/71002713:59
kevinzyoctozepto, morning14:04
kevinzdo we have a procedure to create octavia managment network?14:05
*** zhanglong has quit IRC14:06
*** zhanglong has joined #openstack-kolla14:07
*** dasp has quit IRC14:08
mnasiadkaosmanlicilegi: that's weird, ubuntu is special with netns or what? :)14:10
mnasiadkaosmanlicilegi: if you can just paste neutron logs somewhere - we can analyse14:11
*** zhanglong has quit IRC14:14
*** zhanglong has joined #openstack-kolla14:16
osmanlicilegimnasiadka: last 500 lines of neutron-l3-agent https://paste.ubuntu.com/p/D6PfpWZSvj/14:20
osmanlicilegiI've discovered that all namespaces were lost. after rolling back that change, all came back.14:21
mnasiadkaosmanlicilegi: so that means we need to add /run/netns mount14:21
mnasiadkaosmanlicilegi: do you have an env to check, if changing /run/ bindmount to /run/netns helps?14:22
mnasiadkawell, helps... doesn't break anything :D14:24
osmanlicilegimnasiadka: I'll test it, give me some time14:25
*** sri_ has quit IRC14:25
mnasiadkaosmanlicilegi: I just can't wait to do a revert on each branch...14:25
openstackgerritMerged openstack/kolla stable/train: CentOS 8: Don't force tag in build jobs  https://review.opendev.org/70245214:26
*** abdysn has joined #openstack-kolla14:28
*** abdysn has quit IRC14:28
mnasiadkaosmanlicilegi: but that's really interesting centos doesn't have a problem with that, nor Ubuntu in CI had a problem14:28
mnasiadkaosmanlicilegi: just checked CI jobs result on Ubuntu, worked like charm14:33
*** zhanglong has quit IRC14:38
*** sean-k-mooney has quit IRC14:38
*** kplant has quit IRC14:43
*** kplant has joined #openstack-kolla14:43
mnasiadkayoctozepto: I think it may be related to update of existing routers in neutron14:44
osmanlicilegimnasiadka: reconfigure on the way...14:45
mgoddardosmanlicilegi: do the netns disappear after restarting the container?14:51
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Fix fernet bootstrap and key distribution - follow up  https://review.opendev.org/70708014:51
mnasiadkamgoddard: currently we only know neutron has problems in accessing netns, in router_update and router_delete14:52
mnasiadkamgoddard: and without bindmounting /run/netns to the host, they might be not persistent :)14:52
mgoddardmgoddard mnasiadka hrw egonzalez yoctozepto rafaelweingartne cosmicsound osmanlicilegi14:55
mgoddardmeeting in 5 ^14:55
yoctozepto414:56
yoctozepto:D14:56
mgoddardslowest countdown ever14:56
mgoddardmnasiadka: tripleo also mounts /run/netns14:56
yoctozeptomgoddard: never counted days to see your loved one, did you?14:56
osmanlicilegimnasiadka: seems mounting /run/netns fixes14:56
mgoddardalso /lib/modules:/lib/modules:ro14:57
mnasiadkathat one I have no clue why :)14:57
mgoddardalso looked like they have to create a dummy netns to ensure the directory exists14:57
yoctozeptomnasiadka: weird, it should hit CI hard14:57
yoctozeptohmm14:58
mnasiadkayoctozepto: well, it should on reconfigure, but it's not hitting14:58
*** jbadiapa has quit IRC14:58
mnasiadkaunless we are not restarting neutron on reconfigure14:58
yoctozeptowe are not14:58
yoctozeptoso it must be restarted to fail?14:58
mgoddardovn metadata agent also has those mounts14:58
mnasiadkamaybe we should force reconfigure to restart all docker containers :D14:58
yoctozeptothen upgrades should be hit anyway14:58
mnasiadkamgoddard: yeah, metadata agent also needs access to netns14:59
osmanlicilegihttps://bugs.launchpad.net/kolla-ansible/+bug/186485614:59
openstackLaunchpad bug 1864856 in kolla-ansible "dhcp-agent failed to open netns" [Undecided,New]14:59
hrwmgoddard: thanks14:59
osmanlicilegisame issue14:59
hrwtime to start15:00
mgoddard#startmeeting kolla15:01
openstackMeeting started Wed Feb 26 15:01:04 2020 UTC and is due to finish in 60 minutes.  The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot.15:01
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:01
*** openstack changes topic to " (Meeting topic: kolla)"15:01
mgoddard#topic rollcall15:01
openstackThe meeting name has been set to 'kolla'15:01
*** openstack changes topic to "rollcall (Meeting topic: kolla)"15:01
mgoddard\o15:01
osmanlicilegio/15:01
yoctozeptoo/15:01
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Add /run/netns bindmount to Neutron containers  https://review.opendev.org/71005115:01
hrw /o/15:01
*** TrevorV has joined #openstack-kolla15:01
mnasiadkao/15:01
hrw\o\ /°\15:01
osmanlicilegi\o/\o/\o/15:02
mgoddard#topic agenda15:03
*** openstack changes topic to "agenda (Meeting topic: kolla)"15:03
mgoddard* Roll-call15:03
mgoddard* Announcements15:03
mgoddard* Review action items from last meeting15:03
mgoddard* CI status15:03
mgoddard* Ussuri release planning (kolla & kolla ansible)15:03
mgoddard* Ussuri release planning (kayobe)15:03
mgoddard* Kolla SIG (aka Kolla Klub?) https://etherpad.openstack.org/p/kolla-sig15:03
mgoddard#topic announcements15:03
*** openstack changes topic to "announcements (Meeting topic: kolla)"15:03
mgoddard#info Rocky will move to extended maintenance (EM) soon15:04
hrw24.02 was a date iirc15:04
mgoddardWaiting for final rocky releases then we can bump versions and release our own final15:04
mgoddardAny other announcements?15:05
yoctozeptowe finally tested to-instance network connectivity in CI15:05
yoctozeptothough that did not catch netns failure ;D15:05
mgoddardyeah that's nice15:06
*** lile has joined #openstack-kolla15:06
mgoddard#topic Review action items from last meeting15:06
*** openstack changes topic to "Review action items from last meeting (Meeting topic: kolla)"15:06
osmanlicilegiyoctozepto: if ci catches everything, we couldn't have adventure :]15:06
mgoddardmnasiadka request neutron 14.1.0 in stein UCA15:06
mgoddardyoctozepto to remove kayobe ceph block device labelling support https://storyboard.openstack.org/#!/story/200729515:06
mgoddardjovial[m] to work on custom extension points15:06
mgoddarddougsz to write bug report about nova SSH nproc issue15:07
mgoddardmnasiadka: done?15:07
mnasiadkamgoddard: complained, but they said they have their own testing regime and will take some time15:07
yoctozeptoregime15:07
yoctozeptototalitarian I presume15:07
mnasiadkawhatever that means15:07
mgoddardok15:07
mgoddarddrop ubuntu15:08
yoctozeptoRIP Stein CI15:08
mgoddardyoctozepto: done?15:08
mgoddardyes15:08
mgoddardthanks15:08
yoctozeptoyw15:08
mgoddardjovial[m] is away, probably more of a long term thing15:08
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: OVN Support  https://review.opendev.org/69684115:09
mgoddarddougsz: you wrote the bug report didn't you?15:09
dougszah sorry, still on my todo list, will do it today15:10
mgoddard#action dougsz to write bug report about nova SSH nproc issue15:10
mgoddard#topic CI status15:10
*** openstack changes topic to "CI status (Meeting topic: kolla)"15:11
*** TrevorV has quit IRC15:11
mgoddardLooks green apart from stein ubuntu-binary15:11
*** TrevorV has joined #openstack-kolla15:11
yoctozeptoindeed15:11
mgoddardI saw some weirdness with ubuntu+keystone on master earlier, hopefully it's my patch15:11
mgoddardalthough I doubt it15:12
mgoddard#topic Ussuri release planning (kolla & kolla ansible)15:12
*** openstack changes topic to "Ussuri release planning (kolla & kolla ansible) (Meeting topic: kolla)"15:12
mgoddardI've been working on the centos8 train backport15:12
yoctozeptomgoddard: all the more reason to depreacate ubuntu15:12
mgoddard#link https://review.opendev.org/#/q/topic:bp/centos-rhel-8+status:open+branch:stable/train15:13
mgoddardthanks for all reviews so far15:13
mgoddardlots of approvals15:13
mgoddardlast few:15:13
mgoddardhttps://review.opendev.org/70975715:13
patchbotpatch 709757 - kolla (stable/train) - Switch to python3 in bindep.txt - 3 patch sets15:13
mgoddardhttps://review.opendev.org/70920415:14
patchbotpatch 709204 - kolla (stable/train) - CentOS 8: Use upstream Ceph/master - 4 patch sets15:14
mgoddardhttps://review.opendev.org/70920315:14
patchbotpatch 709203 - kolla (stable/train) - CentOS 8: Remove shellinabox from ironic-conductor - 4 patch sets15:14
*** sean-k-mooney has joined #openstack-kolla15:14
mgoddardhttps://review.opendev.org/70971815:14
patchbotpatch 709718 - kolla-ansible (stable/train) - CI: Use python 3 for local kolla-ansible execution - 2 patch sets15:14
mgoddardhttps://review.opendev.org/70971715:14
patchbotpatch 709717 - kolla-ansible (stable/train) - CI: Move ansible installation & configuration to A... - 1 patch set15:14
mgoddardthose last two need to merge before deploy jobs will pass - I was too lazy to add depends-on15:15
mgoddardand this one will fix ironic jobs: https://review.opendev.org/70975115:16
patchbotpatch 709751 - kolla-ansible (stable/train) - CI: Use upper constraints when installing clients - 2 patch sets15:16
* mgoddard stops begging for reviews15:16
mnasiadkalol15:16
mgoddardWhat other nice ussuri work should we discuss today?15:16
mnasiadkaI spoke with CentOS Storage SIG, it seems in Ussuri we will use Nautilus15:17
mnasiadkaWhich is in a bit of contrary to what Sage said about CentOS 8 and Ceph release support15:17
mnasiadkaWill investigate that topic, since upstream Ceph repo doesn't have Nautilus on el815:18
*** skramaja has quit IRC15:18
mgoddardnot sage adivce15:18
mgoddard*advice15:18
mnasiadkaand CentOS Storage SIG seems to have it working15:18
mnasiadkaquite a nice desync in one company :)15:18
hrwOctopus was not released yet iirc15:18
hrwmnasiadka: which company you mean?15:18
mnasiadkahrw: Red Hat15:19
hrwmnasiadka: Ceph is not RH product15:19
mgoddardthey probably realised someone might want to run ceph on centos 8 before the middle of this year15:19
yoctozeptoprobably15:20
mnasiadkamgoddard: yeah, but still those packages are from CentOS Storage SIG, not Ceph upstream - but those were always built with different deps15:20
mgoddardwe have a patch to switch to nautilus, seems to work15:20
mnasiadkaNow that we don't have ceph-kolla, life should be easier - whatever the release of Ceph we are using15:21
yoctozeptomnasiadka: a bit15:21
yoctozeptostill need to cater for right client libs15:21
yoctozeptomgoddard: reviewed15:22
mnasiadkaok, end of Ceph topic - I just want to make sure no weird bugs will happen and Ceph bug scrubbing team will tell "we don't support this"15:22
mnasiadka;-)15:22
mgoddardwell we can't release with master15:23
mgoddardwe can bump to octopus when available if we choose15:23
mnasiadkawell, Ussuri release is 13th May, Octopus release is 31st March15:24
mgoddardgeneralfuzz: you around?15:24
generalfuzzyes15:25
mgoddardhow is the backend tls work going?15:25
generalfuzzI believe it is progressing nicely. I would like to get another set of reviews on the current patch - https://review.opendev.org/#/c/66451615:26
patchbotpatch 664516 - kolla-ansible - Add support for encrypting backend HAProxy traffic - 20 patch sets15:26
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Fix fernet bootstrap and key distribution - follow up  https://review.opendev.org/70708015:27
mgoddardare you and yongjun bai communicating?15:27
generalfuzzsome15:28
mgoddardI don't know if you've seen the patches coming in from them15:28
mgoddardjust want to make sure everyone's on the same page15:28
mgoddardwe've talked about trying to create common roles for some of this stuff to reduce duplication15:29
generalfuzzI will send a note today. My goal is to have an agreed upon implementation in https://review.opendev.org/#/c/664516. Then we can split out the services15:29
patchbotpatch 664516 - kolla-ansible - Add support for encrypting backend HAProxy traffic - 20 patch sets15:29
mgoddardmakes sense15:29
generalfuzzI'm going to look into a wsgi template generation task next15:29
mgoddardok15:30
mgoddardquick poll: is a generic wsgi/apache config template worth doing?15:30
osmanlicilegi+115:30
mnasiadkaI thought about the same today15:30
yoctozeptoare we doing deprecated mod_wsgi now?15:31
mnasiadkamod_wsgi is deprecated?15:31
yoctozeptomnasiadka: some os services marked it not recommended15:31
mnasiadkayoctozepto: and what is recommended?15:31
yoctozeptomnasiadka: uwsgi is the way forward15:31
osmanlicilegiuwsgi I think15:31
yoctozeptoit actually works in devstack15:31
yoctozepto;p15:32
generalfuzzI was unable to get uwsgi to work with certs15:32
yoctozeptohmm15:32
mnasiadkayoctozepto: devstack is a buzzword, me don't believe15:32
mgoddardand this wasn't mentioned because...15:32
yoctozeptomnasiadka: trust me, I'm core ;p15:32
mnasiadkayoctozepto: maybe I'm old fashioned, but can OpenStack make a statement on what is the direction? because I feel in next release they will say uwsgi is bad, and we should go to this shiny new tool15:33
mgoddardshould we be using uwsgi instead then?15:33
yoctozeptogenerally yes, except for glance which wants to stay with its old eventlet15:33
mgoddardok, we have two separate things here15:34
mnasiadkayoctozepto: and we have a change for glance to use mod_wsgi15:34
generalfuzzI got exceptions when I configured services with uwsgi and defined the certs. I can revisit that today to pinpoint the exceptions15:34
yoctozeptomnasiadka: block it15:35
mgoddard1. backend tls - general pattern, usage, etc15:35
mgoddard2. backend web server15:35
mgoddardif 2 is contentious we can continue with 115:35
yoctozeptoapache can do mod_proxy15:36
mgoddardbut let's not go adding mod_wsgi everywhere if its genuinely deprecated15:36
yoctozeptogeneralfuzz: what broke with tls in uwsgi?15:36
mnasiadkahttps://governance.openstack.org/tc/goals/selected/pike/deploy-api-in-wsgi.html#uwsgi-vs-mod-wsgi15:36
mnasiadka(it's pike - but states devstack done move to uwsgi)15:36
mgoddardI'm fairly sure OSA supports uwsgi15:36
yoctozepto"with the intent that the mod_wsgi support is deleted from devstack in Queens."15:36
openstackgerritMerged openstack/kolla stable/train: CentOS 8: base and openstack-base images  https://review.opendev.org/70953715:36
mgoddard#link https://docs.openstack.org/ansible-role-uwsgi/latest/15:37
yoctozeptoyeah, osa is uwsgi15:37
yoctozeptoor at least to some degree15:37
mgoddardnot in devstack doesn't mean deprecated15:37
generalfuzzyoctozepto: there were python openssl exceptions. I will revisit today15:38
mgoddardgeneralfuzz: would be interesting to see what you changed to get uwsgi going15:39
yoctozeptomgoddard: right but if osa and devstack push towards uwsgi, then mod_wsgi becomes obsolete and can break15:40
mgoddardwhat about tripleo?15:40
yoctozeptoalso see: https://bugs.launchpad.net/neutron/+bug/186441815:41
openstackLaunchpad bug 1864418 in neutron "has wrong with use apache to start neutron api in docker container" [Undecided,New]15:41
yoctozeptothis might be neutron behind mod_wsgi issue15:41
yoctozeptomgoddard: good question15:41
openstackgerritMerged openstack/kolla stable/train: CentOS 8: Update packages in images  https://review.opendev.org/70920215:42
mgoddardlooks like quite a lot of wsgi in tripleo15:42
mgoddardwhich means kolla images probably need to keep apache packages15:42
mgoddard(unless we get them to override)15:42
generalfuzzmgoddard: uwsgi is supported out of the box for nova. I'll need to look at Placement + keystone15:43
mgoddardok, sounds like more research required on wsgi vs. uwsgi15:43
yoctozeptogeneralfuzz: keystone is uwsgi-only in osa15:43
mnasiadkamgoddard: well, in theory we could support both15:43
yoctozeptomhm, tripleo looks mod_wsgi-only15:43
generalfuzzwe may need a hybrid approach15:44
mgoddardmnasiadka: no thanks :)15:44
yoctozeptomgoddard: mnasiadka is right15:44
yoctozeptostuffing both required parts in kolla is nobrainer15:44
mnasiadkamgoddard: I think it will end up like this unfortunately15:44
mgoddardwhy?15:44
yoctozeptoand we might want a phasing out approach15:44
yoctozepto^15:44
yoctozeptoas it may break any day15:44
mnasiadkafor now it works I guess, so it's not critical ;)15:45
mgoddardindeed15:45
yoctozeptoagreed15:45
mnasiadkamore convenient would be to have some common template or role that unifies mod_wsgi configs15:45
mgoddardgeneralfuzz: I would suggest not adding any more mod_wsgi configs for now :)15:45
generalfuzzI will re-look at uwsgi with TLS.15:46
yoctozeptowell, those two kinda contradict each other15:46
mgoddardwell no point in a common mod_wsgi role if we move to uwsgi15:46
*** Trevor_V has joined #openstack-kolla15:46
mgoddardscrew wsgi, I need a whiskey15:46
generalfuzzIs adding ability to execute the container with the "root" user an acceptable solution?15:47
mgoddardlet's try to get backend tls polished and merged with just keystone support15:47
yoctozeptowsgiey15:47
generalfuzzmgoddard: I will remove the nova + placement for now15:47
yoctozeptoit seriously should not be necessary15:47
mnasiadkaaround uwsgi - I just hope uwsgi version between distro is at least a bit consistent, looking at mod_wsgi versions that we have now (and have to use medieval configs due to CentOS)15:48
generalfuzzyoctozepto: how can I have the container run the apache script with sudo from k-a code?15:48
mgoddardon the root user - normally we change it in the container image15:48
mgoddardUSER root15:49
yoctozepto^15:49
mgoddardquestion is whether this presents a transition problem for tripleo or k-a15:49
*** TrevorV has quit IRC15:49
generalfuzzso I should go into docker scripts in kolla as a related change?15:50
mgoddardlooks like tripleo might already use wsgi for now15:50
mgoddardyes15:50
yoctozeptoor eventlet15:50
yoctozeptohard to catch that15:50
mgoddardI'd be interested to see what files the uwsgi config requires for TLS15:50
yoctozeptoas it's just running py script15:50
openstackgerritMerged openstack/kolla-ansible stable/train: CentOS 8: Support variable image tag suffix  https://review.opendev.org/70953415:50
mnasiadkathis looks like the easiest thing on earth: https://uwsgi-docs.readthedocs.io/en/latest/HTTPS.html15:51
mgoddardi.e. if we merge backend tls support for wsgi, could we switch to uwsgi without any change in the user interface (config files)15:51
mnasiadkaSo I'm pretty sure it doesn't work15:51
*** dasp has joined #openstack-kolla15:52
yoctozepto--https 0.0.0.0:8443,foobar.crt,foobar.key15:52
yoctozeptowell, that pretty much explains what tls really is15:52
generalfuzzmgoddard: config files would change, since we would declare cert/key in .conf file15:53
hrwmgoddard: on my server I use nginx to wrap uwsgi with tls15:53
mgoddardgeneralfuzz: that's fine - we control those15:53
mgoddardlooks like it's just a key and cert, same as wsgi15:54
yoctozeptowe always needs this at min15:55
yoctozeptoand then any shim to convert/glue to the required form would be sufficient15:55
yoctozeptolike that ugly haproxy habing key with cert15:55
yoctozeptowho came up with that15:55
mgoddarddoes devstack use mod_uwsgi or uwsgi binary?15:56
ktibimgoddard, kayobe can auto resize lvm thinpool on seed if the disk is increase ?15:56
mgoddardktibi: meeting time15:56
mgoddardok, seems this has taken most of the meeting15:57
mgoddard#topic Ussuri release planning (kayobe)15:57
*** openstack changes topic to "Ussuri release planning (kayobe) (Meeting topic: kolla)"15:57
yoctozeptomgoddard: uwsgi15:57
mnasiadkakayobe as wsgi app?15:57
yoctozeptomgoddard: external binary15:57
mnasiadka(just laughing to continue wsgi topic)15:58
yoctozeptomnasiadka: yes, wsgify kayobe15:58
mgoddardI don't have much to say other than I have more CentOS 8 patches for kayobe - reviews please dougsz & priteau :)15:58
mgoddarduwsgi yoctozepto mnasiadka15:58
dougsz:)15:58
priteauwill do15:58
mgoddardthanks15:58
openstackgerritMarcin Juszkiewicz proposed openstack/kolla stable/train: Bump train versions  https://review.opendev.org/71006715:59
* yoctozepto cannot be deployed as wsgi app under mnasiadka15:59
mgoddardI put together a testing checklist for kayobe & centos 815:59
mgoddardhttps://etherpad.openstack.org/p/kolla-centos815:59
mgoddardplease add to it if you think of anything else15:59
*** TrevorV has joined #openstack-kolla15:59
mgoddardand if you have any time for testing please let me know16:00
mgoddard(no doubt testing & fixing)16:00
mgoddardwe outta time16:00
mgoddardthanks all16:00
*** diurnalist has joined #openstack-kolla16:00
mgoddard#endmeeting16:00
*** openstack changes topic to "Kolla IRC meetings on Wednesdays @ 15:00 UTC - see agenda @ https://goo.gl/OXB0DL | Whiteboard: https://etherpad.openstack.org/p/KollaWhiteBoard | IRC channel is *LOGGED* @ http://goo.gl/3mzZ7b"16:00
openstackMeeting ended Wed Feb 26 16:00:40 2020 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:00
openstackMinutes:        http://eavesdrop.openstack.org/meetings/kolla/2020/kolla.2020-02-26-15.01.html16:00
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/kolla/2020/kolla.2020-02-26-15.01.txt16:00
openstackLog:            http://eavesdrop.openstack.org/meetings/kolla/2020/kolla.2020-02-26-15.01.log.html16:00
yoctozeptothanks mgoddard16:00
openstackgerritMarcin Juszkiewicz proposed openstack/kolla stable/stein: Bump stein versions  https://review.opendev.org/71006816:01
hrwfor rocky we would need to do by hand checks as rocky-em is already on for some components16:02
*** Trevor_V has quit IRC16:03
mgoddardhrw: should we update the script to ignore that tag?16:03
mgoddardktibi: I have resized thinpools before, not sure about automatically by kayobe though16:04
ktibimgoddard, ok. I try to use --wipe-disks but kayobe don't remove lvm partition16:05
hrwmgoddard: on all branches probably16:06
*** lile has quit IRC16:11
priteauktibi: --wipe-disks should clear out all LVM info, have you checked if something on that disk is still mounted?16:12
hrwmgoddard: https://paste.centos.org/view/26c75996 is quick change (rocky branch)16:12
ktibipriteau, it's on seed VM and bifrost container is not created.16:13
*** lile has joined #openstack-kolla16:15
hrwmgoddard: master has -em ignoring. we just not cherrypicked it16:17
mgoddardah16:17
openstackgerritMerged openstack/kolla-ansible stable/train: CentOS 8: Deploy CentOS 8 containers  https://review.opendev.org/70953516:17
hrwguess who wrote it...16:17
hrwcommit f528ad81026526af02610aa3e08200fd53b87ab916:17
hrwAuthor: Mark Goddard <mark@stackhpc.com>16:17
hrwDate:   Mon May 20 19:00:33 2019 +010016:17
hrw    Ignore EM releases in version-check.py16:17
cosmicsoundi enable tls and define own tls in place, and in end i still get a self generate ssl how is this even possible?16:18
yoctozeptospoiler alert was missed16:18
cosmicsoundor is this normal behaviour?16:18
openstackgerritMarcin Juszkiewicz proposed openstack/kolla stable/stein: Ignore EM releases in version-check.py  https://review.opendev.org/71007316:18
yoctozeptono, it's not16:18
yoctozeptoI run external tls and it works fine16:18
openstackgerritMarcin Juszkiewicz proposed openstack/kolla stable/rocky: Ignore EM releases in version-check.py  https://review.opendev.org/71007416:18
cosmicsoundwhat vars you use16:18
cosmicsoundil share mine nnow16:18
cosmicsoundsecond16:18
cosmicsoundhttps://mdb.uhlhost.net/uploads/928d63db84b9ec23/image.png here are globals.yml16:19
hrwmarked Rocky one as RP+116:19
cosmicsoundhaproxy-ca.crt  haproxy-ca-internal.crt  haproxy-internal.pem  haproxy.pem have these in my /etc/kolla/certificates16:20
yoctozeptocosmicsound16:21
yoctozeptohttp://paste.openstack.org/show/790038/16:21
yoctozeptohttp://paste.openstack.org/show/790039/16:22
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Fix fernet bootstrap and key distribution - follow up  https://review.opendev.org/70708016:22
openstackgerritMerged openstack/kolla-ansible stable/train: Add python3-dev[el] to bindep.txt  https://review.opendev.org/70969416:23
cosmicsoundyoctozepto , what was the command to get to the one file .pem16:35
cosmicsoundthats the normal crt with the .key into one .pem?16:36
yoctozeptoyeah, they are concatenated16:36
cosmicsound-----END CERTIFICATE-----16:36
cosmicsound-----BEGIN RSA PRIVATE KEY-----16:36
cosmicsoundi have first cert and then followed by key16:36
cosmicsoundinside nothing elese16:36
cosmicsoundil give it a go16:36
cosmicsoundso no eed for ca_chain]16:36
yoctozeptoI have cert, cert and key16:37
yoctozeptowhole chain to allow for match with root16:37
cosmicsoundnot sure i get this16:38
cosmicsounddo you take your instance public?16:39
cosmicsoundor is just internal tls16:39
cosmicsoundroot = ca authority who made the ssl? in my case is sectigo old comodo16:39
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: OVN Support  https://review.opendev.org/69684116:42
yoctozeptocosmicsound: certs are usually signed by an intermediate16:43
yoctozeptocosmicsound: and only root is trusted16:43
yoctozeptocosmicsound: so omitting intermediary from cert chain may render the connection untrusted16:43
hrwmgoddard: found the reason for mistral-dashboard going backwards...16:43
hrwmgoddard: will discuss with release team16:44
cosmicsoundyes so i need the provider trust chain16:44
cosmicsoundgot it16:44
*** bengates has quit IRC16:50
openstackgerritMarcin Juszkiewicz proposed openstack/kolla stable/train: Bump train versions  https://review.opendev.org/71006716:53
cosmicsoundredeploying 🤞17:05
cosmicsoundgot it into one .pem all17:05
cosmicsoundca chain cert and key17:05
cosmicsoundi am confused a bit with node_custom_config and node_config are they same?17:06
openstackgerritMerged openstack/kolla-ansible stable/train: CI: Move ansible installation & configuration to Ansible  https://review.opendev.org/70971717:06
cosmicsoundnode_custom_config = /etc/kolla/config17:06
cosmicsoundcerts have the {{ode_custom}}/certificates . do i need to put them in /config ?17:07
cosmicsound*node17:07
ktibimgoddard, when bifrost start I can see in the container a process: git-remote-https origin https://opendev.org/openstack/ironic17:10
ktibithe image need to be have internet access ?17:10
mgoddardktibi: unfortunately, yes17:14
ktibimgoddard, hum, any workarround ?17:15
mgoddardktibi: maybe you can pass some --skip-tags to bifrost?17:15
mgoddardor add config to point to local repos17:15
ktibithe playbook install bifrost in the container during the bootstrap ?17:16
mgoddardktibi: we run bifrost install when the container is created17:17
openstackgerritMerged openstack/kolla-ansible stable/train: CI: Use upper constraints when installing clients  https://review.opendev.org/70975117:17
openstackgerritMerged openstack/kolla-ansible stable/train: CI: Use python 3 for local kolla-ansible execution  https://review.opendev.org/70971817:17
mgoddardthen again with some --skip-tags during bootstrap17:17
mgoddardmaybe we are missing some tags to skip?17:17
ktibiwhen you say "is created" it's during the build of the image ?17:18
mgoddardyes17:19
openstackgerritMark Goddard proposed openstack/kolla-ansible stable/train: CentOS 8: Add deploy jobs in CI  https://review.opendev.org/70953617:19
ktibimgoddard, ok I can see a task in bifrost-prep, Download via GIT with a var bifrost_install_sources17:21
ktibihow with kayobe can I override variables in this role ?17:21
cosmicsound+1 yoctozepto17:21
cosmicsoundloving kolla more and more each time i pass one messy one like this17:21
cosmicsoundhttps://mdb.uhlhost.net/uploads/b09146c0dbe1a931/image.png17:21
cosmicsoundTLS works on public domain17:22
mgoddardktibi: https://docs.openstack.org/kayobe/latest/configuration/bifrost.html#custom-configuration17:22
cosmicsoundnow need to solve internal domain naming17:22
ktibimgoddard, but I don't understand why I need to clone ironic repo. because it's a source image ? and not a binary ?17:22
hrwbye17:23
mgoddardktibi: it's just part of bifrost installation17:24
ktibiyes but the installation is during the build, not during the run no ?17:25
ktibilike binary image17:25
ktibimgoddard, because I can see the call to the install playbook in the dockerfile, but why kayobe run again this playbook during the bootstrap of image :/17:30
ktibimgoddard, ok I found, in kolla-ansible it's skip_package_install=true but in the playbook it's when: skip_install is not defined17:34
*** evrardjp has quit IRC17:34
*** evrardjp has joined #openstack-kolla17:35
ktibiI try to add skip_install: true with your solution :) works17:40
*** gfidente is now known as gfidente|afk17:40
* yoctozepto is glad, cosmicsound17:44
* cosmicsound is gratefull for such a community17:45
cosmicsoundThe only one think I did not managed to get going17:46
cosmicsoundwas allocation of ipv4 vip external17:46
cosmicsoundwhile if you remember on prechecks it used to pass the pinng17:46
cosmicsoundin the end no horizon was accessible17:46
cosmicsoundthe bug i opened tried to explain this17:46
yoctozeptocosmicsound: where are you showing the cert being deployed then? ;p17:47
cosmicsoundwell this is made with haproxy off and keepalive17:47
*** ktibi has quit IRC17:48
cosmicsoundand i used main public ip assigned to my bare servers17:48
cosmicsoundI have eno1 public and eno2 internal17:48
cosmicsound172.22.0.1/24 with .22 internal vip and eno1 main ipv4 as external ip yet not VIP since haproxy is off17:49
*** k_mouza has quit IRC17:49
cosmicsoundwith haproxy on it wont work, i added another address set it was same no ping so il debug this more17:50
cosmicsoundat first its netplan causing it17:50
cosmicsoundwill try also bond approach since now its flat17:50
cosmicsoundbridge sorry17:50
cosmicsoundNow i made the main ssl work, will give it another chance17:51
cosmicsoundneed more rtm on neutron17:52
cosmicsoundi know il make a terminal record session17:53
cosmicsoundis easiest to see what is done17:53
*** dougsz has quit IRC17:57
*** diurnalist has quit IRC18:07
*** lile has quit IRC18:11
*** tonythomas has quit IRC18:21
*** diurnalist has joined #openstack-kolla18:25
*** lile has joined #openstack-kolla18:27
*** priteau has quit IRC18:32
*** igordc has joined #openstack-kolla18:37
*** igordc has quit IRC18:43
openstackgerritMerged openstack/kayobe master: CentOS 8: Use ansible_playbook_python for localhost dependencies  https://review.opendev.org/70892118:45
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: OVN Support  https://review.opendev.org/69684118:48
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Fix fernet bootstrap and key distribution - follow up  https://review.opendev.org/70708018:49
osmanlicilegineed a quick review for https://review.opendev.org/#/c/705608/18:49
patchbotpatch 705608 - kolla-ansible - Fixes gnocchi-api script name for Ubuntu/Debian - 1 patch set18:49
mnasiadkaosmanlicilegi: done18:51
osmanlicilegithanks!18:51
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Add /run/netns bindmount to Neutron containers  https://review.opendev.org/71005118:54
cosmicsoundyoctozepto , this one is the answer on my precheks now19:39
cosmicsoundTASK [haproxy : Checking if kolla_internal_vip_address and kolla_external_vip_address are not pingable from any node] ***********************************************19:39
cosmicsoundok: [compute-1] => (item={'address': '172.22.0.22', 'command': 'ping'})19:39
cosmicsoundfailed: [compute-1] (item={'address': '51.91.153.141', 'command': 'ping'}) => {"ansible_loop_var": "item", "changed": false, "cmd": ["ping", "-c", "3", "51.91.153.141"], "delta": "0:00:02.050088", "end": "2020-02-26 19:38:08.288743", "failed_when_result": true, "item": {"address": "51.91.153.141", "command": "ping"}, "rc": 0, "start": "2020-02-2619:39
cosmicsound19:38:06.238655", "stderr": "", "stderr_lines": [], "stdout": "PING 51.91.153.141 (51.91.153.141) 56(84) bytes of data.\n64 bytes from 51.91.153.141: icmp_seq=1 ttl=64 time=0.022 ms\n64 bytes from 51.91.153.141: icmp_seq=2 ttl=64 time=0.077 ms\n64 bytes from 51.91.153.141: icmp_seq=3 ttl=64 time=0.084 ms\n\n--- 51.91.153.141 ping statistics ---\n319:39
cosmicsoundpackets transmitted, 3 received, 0% packet loss, time 2048ms\nrtt min/avg/max/mdev = 0.022/0.061/0.084/0.027 ms", "stdout_lines": ["PING 51.91.153.141 (51.91.153.141) 56(84) bytes of data.", "64 bytes from 51.91.153.141: icmp_seq=1 ttl=64 time=0.022 ms", "64 bytes from 51.91.153.141: icmp_seq=2 ttl=64 time=0.077 ms", "64 bytes from 51.91.153.141:19:39
cosmicsoundicmp_seq=3 ttl=64 time=0.084 ms", "", "--- 51.91.153.141 ping statistics ---", "3 packets transmitted, 3 received, 0% packet loss, time 2048ms", "rtt min/avg/max/mdev = 0.022/0.061/0.084/0.027 ms"]}19:39
cosmicsoundalto the ip its pingable19:39
cosmicsoundits indeed not a floating ip, with a flowting ip this will pass just my dashboard will be gone19:39
yoctozeptoit must *not* be pingable *before* deployment19:39
yoctozeptobecause it will be configured19:40
cosmicsoundright19:40
cosmicsoundbecause indeed this replies to pings19:40
cosmicsoundlast log i showed you it passed just no console anywhere19:41
cosmicsoundwill retry that range19:41
openstackgerritDincer Celik proposed openstack/kolla-ansible stable/train: Use more permissive regex to remove the offending 127.0.1.1  https://review.opendev.org/71012619:55
openstackgerritDincer Celik proposed openstack/kolla-ansible stable/stein: Use more permissive regex to remove the offending 127.0.1.1  https://review.opendev.org/71012719:55
yoctozeptoosmanlicilegi: no rocky? ^20:02
*** diurnalist has quit IRC20:26
*** kozhukalov has quit IRC20:33
*** kozhukalov has joined #openstack-kolla20:36
*** sean-k-mooney has quit IRC20:41
*** diurnalist has joined #openstack-kolla20:41
*** kplant has quit IRC20:49
openstackgerritMerged openstack/kolla stable/stein: Ignore EM releases in version-check.py  https://review.opendev.org/71007321:09
openstackgerritMerged openstack/kolla stable/rocky: Ignore EM releases in version-check.py  https://review.opendev.org/71007421:09
openstackgerritMerged openstack/kolla-ansible master: Fixes gnocchi-api script name for Ubuntu/Debian  https://review.opendev.org/70560821:09
*** priteau has joined #openstack-kolla21:28
*** kozhukalov has quit IRC21:29
*** k_mouza has joined #openstack-kolla21:50
*** k_mouza has quit IRC21:55
*** priteau has quit IRC22:11
*** JangwonLee has quit IRC22:31
*** TrevorV has quit IRC22:38
*** ktibi has joined #openstack-kolla23:04
*** ktibi has quit IRC23:09
*** JangwonLee has joined #openstack-kolla23:11
*** cah_link1 has joined #openstack-kolla23:20
*** cah_link has quit IRC23:23
*** cah_link1 is now known as cah_link23:23
*** JangwonLee has quit IRC23:45
*** JangwonLee has joined #openstack-kolla23:47

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!