*** lile has quit IRC | 00:10 | |
*** cah_link has quit IRC | 00:13 | |
*** diurnalist has quit IRC | 00:35 | |
*** diurnalist has joined #openstack-kolla | 00:37 | |
*** sean-k-mooney has joined #openstack-kolla | 00:39 | |
*** k_mouza has joined #openstack-kolla | 00:45 | |
*** k_mouza has quit IRC | 00:49 | |
*** sean-k-mooney has quit IRC | 00:59 | |
*** diurnalist has quit IRC | 01:11 | |
*** diurnalist has joined #openstack-kolla | 01:18 | |
*** diurnalist has quit IRC | 01:49 | |
*** mrunge has quit IRC | 02:30 | |
*** mrunge has joined #openstack-kolla | 02:32 | |
*** zhanglong has joined #openstack-kolla | 03:13 | |
*** zhanglong has quit IRC | 03:36 | |
*** zhanglong has joined #openstack-kolla | 03:37 | |
openstackgerrit | Hongbin Lu proposed openstack/kolla master: Zun: add zun-cni-daemon image https://review.opendev.org/708273 | 03:48 |
---|---|---|
*** hongbin has joined #openstack-kolla | 03:49 | |
openstackgerrit | Hongbin Lu proposed openstack/kolla-ansible master: [WIP] Zun: Add zun-cni-daemon to compute node https://review.opendev.org/708213 | 04:15 |
*** factor has quit IRC | 04:31 | |
*** factor has joined #openstack-kolla | 04:31 | |
openstackgerrit | Yongjun Bai proposed openstack/kolla-ansible master: WIP:Add support for encrypting glance/heat api https://review.opendev.org/707131 | 04:32 |
openstackgerrit | Hongbin Lu proposed openstack/kolla-ansible master: [WIP] Zun: Add zun-cni-daemon to compute node https://review.opendev.org/708213 | 04:34 |
*** hongbin has quit IRC | 04:39 | |
*** zhanglong has quit IRC | 04:53 | |
*** zhanglong has joined #openstack-kolla | 04:55 | |
openstackgerrit | James Kirsch proposed openstack/kolla-ansible master: Add support for encrypting backend HAProxy traffic https://review.opendev.org/664516 | 05:02 |
*** evrardjp has quit IRC | 05:34 | |
*** evrardjp has joined #openstack-kolla | 05:35 | |
*** skramaja has joined #openstack-kolla | 05:35 | |
*** zhanglong has quit IRC | 05:58 | |
*** zhanglong has joined #openstack-kolla | 06:01 | |
*** sri_ has joined #openstack-kolla | 06:04 | |
*** kozhukalov has joined #openstack-kolla | 06:08 | |
*** zhanglong has quit IRC | 06:20 | |
*** zhanglong has joined #openstack-kolla | 06:22 | |
*** shyamb has joined #openstack-kolla | 06:28 | |
*** tony31 has joined #openstack-kolla | 06:37 | |
*** k_mouza has joined #openstack-kolla | 06:46 | |
*** k_mouza has quit IRC | 06:51 | |
*** cah_link has joined #openstack-kolla | 07:09 | |
*** shyamb has quit IRC | 07:14 | |
yoctozepto | morning | 07:32 |
sri_ | morning | 07:40 |
sri_ | yoctozepto, sorry for silly question, how many controller we required to run 30 to 50 compute nodes with DVR enabled | 07:45 |
yoctozepto | sri_: even one controller might be just fine, not sure if dvr changes anything in that regard; for ha you might still want more, probably 3 for default service placement | 07:49 |
sri_ | yoctozepto, planing to use 3 controller for sure, my workload is very generic, create a bunch of vm and just running schedule snapshots and backups. | 07:52 |
sri_ | yoctozepto, with dvr the network traffic don't need to the controllers | 07:53 |
cosmicsound | good day | 07:54 |
*** dciabrin has joined #openstack-kolla | 07:54 | |
sri_ | I think 3 controller should be ok as you mentioned | 07:55 |
yoctozepto | sri_: it never has to go there, you probably mixed controller with networking node | 07:55 |
sri_ | yoctozepto, yes network with mixed with controller nodes | 07:56 |
cosmicsound | yoctozepto , how can we debug better this designate deploy process in kolla, there are some weird issues here to look into, last time i had in designate_worker another pool id from the one in passwords.yml. now the pool id its good and still say no pool available | 07:59 |
*** bengates has joined #openstack-kolla | 08:02 | |
tony31 | morning | 08:03 |
yoctozepto | cosmicsound: you can run services with debug=True if logs are not satisfactorily detailed | 08:07 |
*** tonythomas has joined #openstack-kolla | 08:11 | |
*** shyamb has joined #openstack-kolla | 08:21 | |
cosmicsound | I have the enable_debug true | 08:22 |
cosmicsound | isnt this same as you mentioned? | 08:22 |
cosmicsound | yoctozepto , i found a issues | 08:24 |
cosmicsound | In designate-sink default.conf we have this | 08:24 |
cosmicsound | https://mdb.uhlhost.net/uploads/fce149d037300410/image.png | 08:24 |
cosmicsound | if you check [handler:nova_fixed] / zone_id and [handler:neutron_floatingip] / zone id | 08:24 |
cosmicsound | They are not filled in | 08:24 |
cosmicsound | Now how do I debug this why is not filled, could it be a error in playbook? | 08:25 |
cosmicsound | same issues we debated last time with yankcrime | 08:25 |
mnasiadka | cosmicsound: do you see this? https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/designate/templates/designate.conf.j2#L66 | 08:34 |
mnasiadka | we have no automation to fill in the zone-id, you need to do it by yourself in /etc/kolla/config/designate.conf | 08:34 |
cosmicsound | its missing only 2 locations in fact in designate-sink | 08:36 |
cosmicsound | the rest of the values are filled in | 08:37 |
cosmicsound | it helped me last time by adding the values in designate-sink.conf | 08:37 |
cosmicsound | and reconfigure testing it now as we speak | 08:37 |
mnasiadka | cosmicsound: as I said, kolla-ansible will leave it blank, you need to configure it somewhere in /etc/kolla/config | 08:42 |
cosmicsound | right | 08:51 |
cosmicsound | editing the template also helps | 08:51 |
cosmicsound | or il try | 08:51 |
openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: Fix fernet bootstrap and key distribution - follow up https://review.opendev.org/707080 | 08:52 |
*** shyamb has quit IRC | 08:55 | |
mgoddard | morning | 09:01 |
tony31 | morning | 09:06 |
*** lennyb has quit IRC | 09:06 | |
*** lennyb has joined #openstack-kolla | 09:07 | |
tony31 | when I run "kayobe overcloud host configure" it tries to run some variables as commands once it finishes. I only noticed this start today - I think I borked the control host 🙈 | 09:07 |
tony31 | `(kayobe) [cv-user@juc-kach1-prd kayobe]$ hilosipip_interface: bond2bash: hilosipip_interface:: command not found` | 09:08 |
yoctozepto | bond2bash :D | 09:08 |
tony31 | :) | 09:08 |
yoctozepto | Bond, Bash Bond | 09:09 |
yoctozepto | ;-) | 09:09 |
tony31 | usually it's bin bash | 09:09 |
tony31 | 🗑️ | 09:09 |
well100 | Morning | 09:15 |
*** dougsz has joined #openstack-kolla | 09:15 | |
openstackgerrit | Mark Goddard proposed openstack/kolla stable/train: CentOS 8: base and openstack-base images https://review.opendev.org/709537 | 09:17 |
openstackgerrit | Mark Goddard proposed openstack/kolla stable/train: CentOS 8: Update packages in images https://review.opendev.org/709202 | 09:17 |
well100 | mgoddard i have a doing to my patch? | 09:19 |
*** ktibi has joined #openstack-kolla | 09:20 | |
yoctozepto | well100: yeah, there is review/testing progress | 09:22 |
well100 | yoctozepto: okay fine im wait for instruciton :-) | 09:28 |
openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/train: Python 3: Use distro_python_version for WSGI python_path https://review.opendev.org/709713 | 09:34 |
hrw | well100: https://docs.openstack.org/tripleo-docs/latest/contributor/contributions.html | 09:39 |
hrw | ops | 09:40 |
hrw | well100: I meant https://wiki.openstack.org/wiki/How_To_Contribute | 09:40 |
openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: CI: CentOS 8: Enable TLS on core scenario https://review.opendev.org/709994 | 09:44 |
mgoddard | well100: hi. Are you asking what needs to be done to your patch? | 09:45 |
openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: DNM: Testing cloudkitty in master https://review.opendev.org/681555 | 09:46 |
openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: Fix fernet bootstrap and key distribution - follow up https://review.opendev.org/707080 | 09:46 |
openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: DNM: Testing cloudkitty in master https://review.opendev.org/681555 | 09:47 |
yoctozepto | hrw: sending folks to tripleo? HOW DARE YOU :D | 09:49 |
mgoddard | well100: yoctozepto has posted some comments on your patch | 09:50 |
mgoddard | (assume we are talking about https://review.opendev.org/#/c/707379) | 09:50 |
patchbot | patch 707379 - kolla-ansible - Cloudkitty cant not conncet to Auth - 2 patch sets | 09:50 |
hrw | can't not connect? | 09:53 |
well100 | mgoddard: should i make a new patch without v3? | 10:04 |
openstackgerrit | Alfredo Moralejo proposed openstack/kolla master: Use StorageSIG repos for Ceph in CentOS8 https://review.opendev.org/707338 | 10:06 |
openstackgerrit | Michal Nasiadka proposed openstack/kayobe master: Allow setting pip_proxy https://review.opendev.org/709580 | 10:08 |
openstackgerrit | Michal Nasiadka proposed openstack/kayobe master: Allow setting pip_proxy https://review.opendev.org/709580 | 10:10 |
tony31 | hi kayobe guys | 10:11 |
tony31 | I managed to trace back this problem which I am having at the moment. I am using one physical node for 2 roles: compute and storage. I found that the tunnel network is not being set on the compute node any longer. Not sure how long this issue has been there. But if I remove the storage role from this node then it successfully gets the tunnel | 10:12 |
tony31 | network | 10:12 |
openstackgerrit | Mark Goddard proposed openstack/kolla master: CentOS 8: Actually disable EPEL, and epel-modular https://review.opendev.org/710003 | 10:12 |
mgoddard | hi tony31, I see. It is normally best to only have one top level role per node | 10:13 |
mgoddard | due to the way we build our lists of network interfaces etc | 10:13 |
tony31 | fair enough :) | 10:13 |
mgoddard | I'd suggest using compute, and adding storage bits as necessary | 10:13 |
tony31 | trying to make the most of the hardware I have - I'll try and spin up a storage VM on another hypervisor | 10:14 |
tony31 | thanks for the advice on it | 10:14 |
mgoddard | tony31: you could add the compute group to kolla_overcloud_inventory_storage_groups | 10:15 |
openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: Fix fernet bootstrap and key distribution - follow up https://review.opendev.org/707080 | 10:15 |
tony31 | hmm | 10:15 |
mgoddard | and add networks to compute_extra_network_interfaces as necessary | 10:15 |
tony31 | is that only in the venv ? | 10:16 |
tony31 | `$ grep -rni -e "kolla_overcloud_inventory_storage_groups"venvs/kayobe/share/kayobe/ansible/group_vars/all/kolla:275:kolla_overcloud_inventory_storage_groups:venvs/kayobe/share/kayobe/ansible/group_vars/all/kolla:298: "{{ kolla_overcloud_inventory_storage_groups }}"` | 10:16 |
mgoddard | tony31: we don't include it in etc/kayobe/kolla.yml but it seems to be required quite frequently so maybe we should add it | 10:17 |
mgoddard | tony31: but in general you can add arbitrary variables to the files in etc/kayobe/*.yml | 10:17 |
mgoddard | we just include the most common ones | 10:17 |
tony31 | Would it work the other way around? if I add `storage_extra_network_interfaces:` and list the tunnel network there? | 10:18 |
*** hjensas has quit IRC | 10:18 | |
*** k_mouza has joined #openstack-kolla | 10:21 | |
*** hjensas has joined #openstack-kolla | 10:21 | |
*** k_mouza has quit IRC | 10:21 | |
*** k_mouza has joined #openstack-kolla | 10:21 | |
openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: DNM: Test swift s3api https://review.opendev.org/709730 | 10:24 |
tony31 | mgoddard - I added `storage_extra_network_interfaces:` and set the tunnel interface there (in a host_vars) and now when I do a host configure, it is including the tunnel network. This is just for testing at the moment. I'm still pulling the pieces together. | 10:27 |
tony31 | i just needed a storage node in the same area that the storage is located so that the build is fast - iscsi over a VPN tunnel was a bit slow (for glance building instances) | 10:28 |
tony31 | :) | 10:28 |
openstackgerrit | Mark Goddard proposed openstack/kolla master: CentOS 8: Actually disable EPEL, and epel-modular https://review.opendev.org/710003 | 10:29 |
*** skramaja has quit IRC | 10:30 | |
openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/train: CI: Use upper constraints when installing clients https://review.opendev.org/709751 | 10:34 |
*** tony31 has quit IRC | 10:35 | |
hrw | bbl | 10:37 |
openstackgerrit | yatin proposed openstack/kolla master: Build collectd image for CentOS8 https://review.opendev.org/709701 | 10:41 |
openstackgerrit | yatin proposed openstack/kolla master: Build collectd image for CentOS8 https://review.opendev.org/709701 | 10:41 |
openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: OVN Support https://review.opendev.org/696841 | 10:41 |
openstackgerrit | yatin proposed openstack/kolla master: Build collectd image for CentOS8 https://review.opendev.org/709701 | 10:42 |
*** sean-k-mooney has joined #openstack-kolla | 10:45 | |
openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: Fix fernet bootstrap and key distribution - follow up https://review.opendev.org/707080 | 10:52 |
openstackgerrit | Merged openstack/kolla stable/train: vitrage-base: honor distro_python_version https://review.opendev.org/709553 | 10:57 |
openstackgerrit | Merged openstack/kolla-ansible master: Fix RabbitMQ hostname address resolution precheck https://review.opendev.org/707892 | 10:57 |
*** priteau has joined #openstack-kolla | 11:01 | |
well100 | i have question to rdns :-) | 11:09 |
well100 | how do i create the dns zone for it to work | 11:10 |
openstackgerrit | Merged openstack/kolla-ansible stable/train: Allow to override external network params in init-runonce https://review.opendev.org/709671 | 11:21 |
openstackgerrit | Merged openstack/kolla-ansible master: CI: Fix TLS upgrade test https://review.opendev.org/709808 | 11:22 |
*** shyamb has joined #openstack-kolla | 11:27 | |
*** shyamb has quit IRC | 11:45 | |
openstackgerrit | Mark Goddard proposed openstack/kolla stable/train: CentOS 8: Remove shellinabox from ironic-conductor https://review.opendev.org/709203 | 11:59 |
openstackgerrit | Mark Goddard proposed openstack/kolla stable/train: CentOS 8: Use upstream Ceph/master https://review.opendev.org/709204 | 11:59 |
openstackgerrit | Mark Goddard proposed openstack/kolla stable/train: CentOS 8: Allow SSH access to keystone_ssh and nova_ssh https://review.opendev.org/709205 | 11:59 |
openstackgerrit | Mark Goddard proposed openstack/kolla stable/train: CentOS 8: Relax ironic iPXE EFI bootloader location https://review.opendev.org/709207 | 11:59 |
openstackgerrit | Mark Goddard proposed openstack/kolla stable/train: Switch to python3 in bindep.txt https://review.opendev.org/709757 | 11:59 |
openstackgerrit | Mark Goddard proposed openstack/kolla stable/train: CentOS 8: Add deploy jobs in CI https://review.opendev.org/709538 | 11:59 |
mgoddard | yoctozepto, mnasiadka, osmanlicilegi: https://review.opendev.org/#/q/topic:bp/centos-rhel-8+status:open+branch:stable/train | 12:00 |
*** shyamb has joined #openstack-kolla | 12:08 | |
openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: OVN Support https://review.opendev.org/696841 | 12:22 |
openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: CI: Add addressing on external network https://review.opendev.org/709362 | 12:27 |
openstackgerrit | Merged openstack/kolla-ansible stable/train: Python 3: Use distro_python_version for monasca agent CA file https://review.opendev.org/709711 | 12:31 |
openstackgerrit | Merged openstack/kolla-ansible stable/train: Remove unused python path calculation from vmtp https://review.opendev.org/709712 | 12:31 |
openstackgerrit | Merged openstack/kolla-ansible stable/train: Use local python interpreter for keystone cron generator https://review.opendev.org/709714 | 12:31 |
openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: Fix fernet bootstrap and key distribution - follow up https://review.opendev.org/707080 | 12:34 |
openstackgerrit | Merged openstack/kolla-ansible stable/train: Support python 3 in kolla-ansible script https://review.opendev.org/709716 | 12:36 |
ktibi | Hello guys, I add a param in kayobe for the registry password. But I can see in the password.yml of kolla, the password is set to null. How can I fix that ? Need to edit manualy the password.yml ? | 12:36 |
ktibi | or maybe I can regenerate the file :/ | 12:39 |
mgoddard | ktibi: I'd suggest using this: https://docs.openstack.org/kayobe/latest/configuration/kolla-ansible.html#configuring-custom-passwords | 12:43 |
mgoddard | maybe we need to add that to the defaults? | 12:43 |
ktibi | ok strange because I have set docker_registry_password (I can see it in global of kolla) but when kayobe check if passwords.yml is correct, kayobe override and add None to the password | 12:44 |
ktibi | mgoddard, ok because kayobe use "{{ temp_path }}/passwords.yml" and I think I need to modify this file :p | 12:46 |
ktibi | ok found, need ot modify etc/kayobe/kolla/password ;) | 12:49 |
*** kplant has joined #openstack-kolla | 12:57 | |
openstackgerrit | Mark Goddard proposed openstack/kolla master: Actually disable EPEL, and epel-modular https://review.opendev.org/710003 | 12:58 |
openstackgerrit | Mark Goddard proposed openstack/kolla master: Throw TypeError in repository enable/disable macros https://review.opendev.org/710027 | 12:58 |
openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: OVN Support https://review.opendev.org/696841 | 12:59 |
openstackgerrit | Michal Nasiadka proposed openstack/kayobe master: Add framework to deploy user-defined containers on seed https://review.opendev.org/709689 | 13:01 |
openstackgerrit | Piotr Rabiega proposed openstack/kolla master: Add collectd-pcie-errors package for PCIe Errors (C7 only) https://review.opendev.org/710029 | 13:06 |
openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: Fix fernet bootstrap and key distribution - follow up https://review.opendev.org/707080 | 13:06 |
well100 | ptr reverse dns can you help me? | 13:07 |
cosmicsound | well100 , what is wrong | 13:10 |
cosmicsound | i work also now on designate | 13:10 |
well100 | do I have to create the ptr zone, if so where do I have to create it? | 13:11 |
cosmicsound | you can do it in /etc/hosts locally or with you provider, not sure how your domain is routed | 13:13 |
cosmicsound | i use for example a dns in cpanel now, that i forward to my server where i make the reverse dns | 13:13 |
cosmicsound | and then i add it locally in /etc/hosts | 13:13 |
cosmicsound | to reflect the ip of the main bind9 host in designate | 13:14 |
cosmicsound | in my case that is dns.uhlhost.net if you do a host dns.uhlhost.net you see it points to my designate ip for this. | 13:14 |
cosmicsound | not sure if this helps | 13:14 |
cosmicsound | also if you have more ips your provider should allow you to change the reverse records, once ofcourse a dns points to them | 13:15 |
cosmicsound | or maybe using the bind templates | 13:15 |
cosmicsound | you could change there all ptr records | 13:15 |
*** skramaja has joined #openstack-kolla | 13:16 | |
well100 | no | 13:26 |
well100 | I want designate to create the PTR records of fixed and floating ips | 13:27 |
openstackgerrit | Merged openstack/kolla-ansible stable/train: Python 3: Use distro_python_version for WSGI python_path https://review.opendev.org/709713 | 13:35 |
osmanlicilegi | mnasiadka: after merging https://review.opendev.org/#/c/707375/ I've discovered ubuntu cannot manage network namespaces anymore. should be same on debian too. | 13:38 |
patchbot | patch 707375 - kolla-ansible - Change /run bind mount for neutron/openvswitch (MERGED) - 4 patch sets | 13:38 |
osmanlicilegi | seems /run rules ubuntu | 13:38 |
osmanlicilegi | I'll raise a bug report soon | 13:39 |
yoctozepto | osmanlicilegi: dang | 13:41 |
yoctozepto | osmanlicilegi: that's odd because it is passing in CI | 13:42 |
yoctozepto | wonder which part broke there | 13:42 |
openstackgerrit | Yongjun Bai proposed openstack/kolla-ansible master: WIP:support for separate admin vip address and admin fqdn https://review.opendev.org/710036 | 13:42 |
*** diurnalist has joined #openstack-kolla | 13:44 | |
openstackgerrit | Merged openstack/kolla stable/train: Remove tgtd and scsi-target-utils support in CentOS/RHEL 8 https://review.opendev.org/702421 | 13:48 |
*** diurnalist has quit IRC | 13:48 | |
mgoddard | :( ubuntu deploy jobs seem broken on master | 13:50 |
*** shyamb has quit IRC | 13:51 | |
yoctozepto | where? how? | 13:53 |
mgoddard | possible it's caused by dropping epel :) | 13:54 |
mgoddard | it's probably fine | 13:54 |
mgoddard | hold up that doesn't make sense on ubuntu! | 13:55 |
mgoddard | wake up mgoddard | 13:55 |
openstackgerrit | Mark Goddard proposed openstack/kolla master: Actually disable EPEL, and epel-modular https://review.opendev.org/710003 | 13:59 |
openstackgerrit | Mark Goddard proposed openstack/kolla master: Throw TypeError in repository enable/disable macros https://review.opendev.org/710027 | 13:59 |
kevinz | yoctozepto, morning | 14:04 |
kevinz | do we have a procedure to create octavia managment network? | 14:05 |
*** zhanglong has quit IRC | 14:06 | |
*** zhanglong has joined #openstack-kolla | 14:07 | |
*** dasp has quit IRC | 14:08 | |
mnasiadka | osmanlicilegi: that's weird, ubuntu is special with netns or what? :) | 14:10 |
mnasiadka | osmanlicilegi: if you can just paste neutron logs somewhere - we can analyse | 14:11 |
*** zhanglong has quit IRC | 14:14 | |
*** zhanglong has joined #openstack-kolla | 14:16 | |
osmanlicilegi | mnasiadka: last 500 lines of neutron-l3-agent https://paste.ubuntu.com/p/D6PfpWZSvj/ | 14:20 |
osmanlicilegi | I've discovered that all namespaces were lost. after rolling back that change, all came back. | 14:21 |
mnasiadka | osmanlicilegi: so that means we need to add /run/netns mount | 14:21 |
mnasiadka | osmanlicilegi: do you have an env to check, if changing /run/ bindmount to /run/netns helps? | 14:22 |
mnasiadka | well, helps... doesn't break anything :D | 14:24 |
osmanlicilegi | mnasiadka: I'll test it, give me some time | 14:25 |
*** sri_ has quit IRC | 14:25 | |
mnasiadka | osmanlicilegi: I just can't wait to do a revert on each branch... | 14:25 |
openstackgerrit | Merged openstack/kolla stable/train: CentOS 8: Don't force tag in build jobs https://review.opendev.org/702452 | 14:26 |
*** abdysn has joined #openstack-kolla | 14:28 | |
*** abdysn has quit IRC | 14:28 | |
mnasiadka | osmanlicilegi: but that's really interesting centos doesn't have a problem with that, nor Ubuntu in CI had a problem | 14:28 |
mnasiadka | osmanlicilegi: just checked CI jobs result on Ubuntu, worked like charm | 14:33 |
*** zhanglong has quit IRC | 14:38 | |
*** sean-k-mooney has quit IRC | 14:38 | |
*** kplant has quit IRC | 14:43 | |
*** kplant has joined #openstack-kolla | 14:43 | |
mnasiadka | yoctozepto: I think it may be related to update of existing routers in neutron | 14:44 |
osmanlicilegi | mnasiadka: reconfigure on the way... | 14:45 |
mgoddard | osmanlicilegi: do the netns disappear after restarting the container? | 14:51 |
openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: Fix fernet bootstrap and key distribution - follow up https://review.opendev.org/707080 | 14:51 |
mnasiadka | mgoddard: currently we only know neutron has problems in accessing netns, in router_update and router_delete | 14:52 |
mnasiadka | mgoddard: and without bindmounting /run/netns to the host, they might be not persistent :) | 14:52 |
mgoddard | mgoddard mnasiadka hrw egonzalez yoctozepto rafaelweingartne cosmicsound osmanlicilegi | 14:55 |
mgoddard | meeting in 5 ^ | 14:55 |
yoctozepto | 4 | 14:56 |
yoctozepto | :D | 14:56 |
mgoddard | slowest countdown ever | 14:56 |
mgoddard | mnasiadka: tripleo also mounts /run/netns | 14:56 |
yoctozepto | mgoddard: never counted days to see your loved one, did you? | 14:56 |
osmanlicilegi | mnasiadka: seems mounting /run/netns fixes | 14:56 |
mgoddard | also /lib/modules:/lib/modules:ro | 14:57 |
mnasiadka | that one I have no clue why :) | 14:57 |
mgoddard | also looked like they have to create a dummy netns to ensure the directory exists | 14:57 |
yoctozepto | mnasiadka: weird, it should hit CI hard | 14:57 |
yoctozepto | hmm | 14:58 |
mnasiadka | yoctozepto: well, it should on reconfigure, but it's not hitting | 14:58 |
*** jbadiapa has quit IRC | 14:58 | |
mnasiadka | unless we are not restarting neutron on reconfigure | 14:58 |
yoctozepto | we are not | 14:58 |
yoctozepto | so it must be restarted to fail? | 14:58 |
mgoddard | ovn metadata agent also has those mounts | 14:58 |
mnasiadka | maybe we should force reconfigure to restart all docker containers :D | 14:58 |
yoctozepto | then upgrades should be hit anyway | 14:58 |
mnasiadka | mgoddard: yeah, metadata agent also needs access to netns | 14:59 |
osmanlicilegi | https://bugs.launchpad.net/kolla-ansible/+bug/1864856 | 14:59 |
openstack | Launchpad bug 1864856 in kolla-ansible "dhcp-agent failed to open netns" [Undecided,New] | 14:59 |
hrw | mgoddard: thanks | 14:59 |
osmanlicilegi | same issue | 14:59 |
hrw | time to start | 15:00 |
mgoddard | #startmeeting kolla | 15:01 |
openstack | Meeting started Wed Feb 26 15:01:04 2020 UTC and is due to finish in 60 minutes. The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:01 |
openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:01 |
*** openstack changes topic to " (Meeting topic: kolla)" | 15:01 | |
mgoddard | #topic rollcall | 15:01 |
openstack | The meeting name has been set to 'kolla' | 15:01 |
*** openstack changes topic to "rollcall (Meeting topic: kolla)" | 15:01 | |
mgoddard | \o | 15:01 |
osmanlicilegi | o/ | 15:01 |
yoctozepto | o/ | 15:01 |
openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: Add /run/netns bindmount to Neutron containers https://review.opendev.org/710051 | 15:01 |
hrw | /o/ | 15:01 |
*** TrevorV has joined #openstack-kolla | 15:01 | |
mnasiadka | o/ | 15:01 |
hrw | \o\ /°\ | 15:01 |
osmanlicilegi | \o/\o/\o/ | 15:02 |
mgoddard | #topic agenda | 15:03 |
*** openstack changes topic to "agenda (Meeting topic: kolla)" | 15:03 | |
mgoddard | * Roll-call | 15:03 |
mgoddard | * Announcements | 15:03 |
mgoddard | * Review action items from last meeting | 15:03 |
mgoddard | * CI status | 15:03 |
mgoddard | * Ussuri release planning (kolla & kolla ansible) | 15:03 |
mgoddard | * Ussuri release planning (kayobe) | 15:03 |
mgoddard | * Kolla SIG (aka Kolla Klub?) https://etherpad.openstack.org/p/kolla-sig | 15:03 |
mgoddard | #topic announcements | 15:03 |
*** openstack changes topic to "announcements (Meeting topic: kolla)" | 15:03 | |
mgoddard | #info Rocky will move to extended maintenance (EM) soon | 15:04 |
hrw | 24.02 was a date iirc | 15:04 |
mgoddard | Waiting for final rocky releases then we can bump versions and release our own final | 15:04 |
mgoddard | Any other announcements? | 15:05 |
yoctozepto | we finally tested to-instance network connectivity in CI | 15:05 |
yoctozepto | though that did not catch netns failure ;D | 15:05 |
mgoddard | yeah that's nice | 15:06 |
*** lile has joined #openstack-kolla | 15:06 | |
mgoddard | #topic Review action items from last meeting | 15:06 |
*** openstack changes topic to "Review action items from last meeting (Meeting topic: kolla)" | 15:06 | |
osmanlicilegi | yoctozepto: if ci catches everything, we couldn't have adventure :] | 15:06 |
mgoddard | mnasiadka request neutron 14.1.0 in stein UCA | 15:06 |
mgoddard | yoctozepto to remove kayobe ceph block device labelling support https://storyboard.openstack.org/#!/story/2007295 | 15:06 |
mgoddard | jovial[m] to work on custom extension points | 15:06 |
mgoddard | dougsz to write bug report about nova SSH nproc issue | 15:07 |
mgoddard | mnasiadka: done? | 15:07 |
mnasiadka | mgoddard: complained, but they said they have their own testing regime and will take some time | 15:07 |
yoctozepto | regime | 15:07 |
yoctozepto | totalitarian I presume | 15:07 |
mnasiadka | whatever that means | 15:07 |
mgoddard | ok | 15:07 |
mgoddard | drop ubuntu | 15:08 |
yoctozepto | RIP Stein CI | 15:08 |
mgoddard | yoctozepto: done? | 15:08 |
mgoddard | yes | 15:08 |
mgoddard | thanks | 15:08 |
yoctozepto | yw | 15:08 |
mgoddard | jovial[m] is away, probably more of a long term thing | 15:08 |
openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: OVN Support https://review.opendev.org/696841 | 15:09 |
mgoddard | dougsz: you wrote the bug report didn't you? | 15:09 |
dougsz | ah sorry, still on my todo list, will do it today | 15:10 |
mgoddard | #action dougsz to write bug report about nova SSH nproc issue | 15:10 |
mgoddard | #topic CI status | 15:10 |
*** openstack changes topic to "CI status (Meeting topic: kolla)" | 15:11 | |
*** TrevorV has quit IRC | 15:11 | |
mgoddard | Looks green apart from stein ubuntu-binary | 15:11 |
*** TrevorV has joined #openstack-kolla | 15:11 | |
yoctozepto | indeed | 15:11 |
mgoddard | I saw some weirdness with ubuntu+keystone on master earlier, hopefully it's my patch | 15:11 |
mgoddard | although I doubt it | 15:12 |
mgoddard | #topic Ussuri release planning (kolla & kolla ansible) | 15:12 |
*** openstack changes topic to "Ussuri release planning (kolla & kolla ansible) (Meeting topic: kolla)" | 15:12 | |
mgoddard | I've been working on the centos8 train backport | 15:12 |
yoctozepto | mgoddard: all the more reason to depreacate ubuntu | 15:12 |
mgoddard | #link https://review.opendev.org/#/q/topic:bp/centos-rhel-8+status:open+branch:stable/train | 15:13 |
mgoddard | thanks for all reviews so far | 15:13 |
mgoddard | lots of approvals | 15:13 |
mgoddard | last few: | 15:13 |
mgoddard | https://review.opendev.org/709757 | 15:13 |
patchbot | patch 709757 - kolla (stable/train) - Switch to python3 in bindep.txt - 3 patch sets | 15:13 |
mgoddard | https://review.opendev.org/709204 | 15:14 |
patchbot | patch 709204 - kolla (stable/train) - CentOS 8: Use upstream Ceph/master - 4 patch sets | 15:14 |
mgoddard | https://review.opendev.org/709203 | 15:14 |
patchbot | patch 709203 - kolla (stable/train) - CentOS 8: Remove shellinabox from ironic-conductor - 4 patch sets | 15:14 |
*** sean-k-mooney has joined #openstack-kolla | 15:14 | |
mgoddard | https://review.opendev.org/709718 | 15:14 |
patchbot | patch 709718 - kolla-ansible (stable/train) - CI: Use python 3 for local kolla-ansible execution - 2 patch sets | 15:14 |
mgoddard | https://review.opendev.org/709717 | 15:14 |
patchbot | patch 709717 - kolla-ansible (stable/train) - CI: Move ansible installation & configuration to A... - 1 patch set | 15:14 |
mgoddard | those last two need to merge before deploy jobs will pass - I was too lazy to add depends-on | 15:15 |
mgoddard | and this one will fix ironic jobs: https://review.opendev.org/709751 | 15:16 |
patchbot | patch 709751 - kolla-ansible (stable/train) - CI: Use upper constraints when installing clients - 2 patch sets | 15:16 |
* mgoddard stops begging for reviews | 15:16 | |
mnasiadka | lol | 15:16 |
mgoddard | What other nice ussuri work should we discuss today? | 15:16 |
mnasiadka | I spoke with CentOS Storage SIG, it seems in Ussuri we will use Nautilus | 15:17 |
mnasiadka | Which is in a bit of contrary to what Sage said about CentOS 8 and Ceph release support | 15:17 |
mnasiadka | Will investigate that topic, since upstream Ceph repo doesn't have Nautilus on el8 | 15:18 |
*** skramaja has quit IRC | 15:18 | |
mgoddard | not sage adivce | 15:18 |
mgoddard | *advice | 15:18 |
mnasiadka | and CentOS Storage SIG seems to have it working | 15:18 |
mnasiadka | quite a nice desync in one company :) | 15:18 |
hrw | Octopus was not released yet iirc | 15:18 |
hrw | mnasiadka: which company you mean? | 15:18 |
mnasiadka | hrw: Red Hat | 15:19 |
hrw | mnasiadka: Ceph is not RH product | 15:19 |
mgoddard | they probably realised someone might want to run ceph on centos 8 before the middle of this year | 15:19 |
yoctozepto | probably | 15:20 |
mnasiadka | mgoddard: yeah, but still those packages are from CentOS Storage SIG, not Ceph upstream - but those were always built with different deps | 15:20 |
mgoddard | we have a patch to switch to nautilus, seems to work | 15:20 |
mnasiadka | Now that we don't have ceph-kolla, life should be easier - whatever the release of Ceph we are using | 15:21 |
yoctozepto | mnasiadka: a bit | 15:21 |
yoctozepto | still need to cater for right client libs | 15:21 |
yoctozepto | mgoddard: reviewed | 15:22 |
mnasiadka | ok, end of Ceph topic - I just want to make sure no weird bugs will happen and Ceph bug scrubbing team will tell "we don't support this" | 15:22 |
mnasiadka | ;-) | 15:22 |
mgoddard | well we can't release with master | 15:23 |
mgoddard | we can bump to octopus when available if we choose | 15:23 |
mnasiadka | well, Ussuri release is 13th May, Octopus release is 31st March | 15:24 |
mgoddard | generalfuzz: you around? | 15:24 |
generalfuzz | yes | 15:25 |
mgoddard | how is the backend tls work going? | 15:25 |
generalfuzz | I believe it is progressing nicely. I would like to get another set of reviews on the current patch - https://review.opendev.org/#/c/664516 | 15:26 |
patchbot | patch 664516 - kolla-ansible - Add support for encrypting backend HAProxy traffic - 20 patch sets | 15:26 |
openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: Fix fernet bootstrap and key distribution - follow up https://review.opendev.org/707080 | 15:27 |
mgoddard | are you and yongjun bai communicating? | 15:27 |
generalfuzz | some | 15:28 |
mgoddard | I don't know if you've seen the patches coming in from them | 15:28 |
mgoddard | just want to make sure everyone's on the same page | 15:28 |
mgoddard | we've talked about trying to create common roles for some of this stuff to reduce duplication | 15:29 |
generalfuzz | I will send a note today. My goal is to have an agreed upon implementation in https://review.opendev.org/#/c/664516. Then we can split out the services | 15:29 |
patchbot | patch 664516 - kolla-ansible - Add support for encrypting backend HAProxy traffic - 20 patch sets | 15:29 |
mgoddard | makes sense | 15:29 |
generalfuzz | I'm going to look into a wsgi template generation task next | 15:29 |
mgoddard | ok | 15:30 |
mgoddard | quick poll: is a generic wsgi/apache config template worth doing? | 15:30 |
osmanlicilegi | +1 | 15:30 |
mnasiadka | I thought about the same today | 15:30 |
yoctozepto | are we doing deprecated mod_wsgi now? | 15:31 |
mnasiadka | mod_wsgi is deprecated? | 15:31 |
yoctozepto | mnasiadka: some os services marked it not recommended | 15:31 |
mnasiadka | yoctozepto: and what is recommended? | 15:31 |
yoctozepto | mnasiadka: uwsgi is the way forward | 15:31 |
osmanlicilegi | uwsgi I think | 15:31 |
yoctozepto | it actually works in devstack | 15:31 |
yoctozepto | ;p | 15:32 |
generalfuzz | I was unable to get uwsgi to work with certs | 15:32 |
yoctozepto | hmm | 15:32 |
mnasiadka | yoctozepto: devstack is a buzzword, me don't believe | 15:32 |
mgoddard | and this wasn't mentioned because... | 15:32 |
yoctozepto | mnasiadka: trust me, I'm core ;p | 15:32 |
mnasiadka | yoctozepto: maybe I'm old fashioned, but can OpenStack make a statement on what is the direction? because I feel in next release they will say uwsgi is bad, and we should go to this shiny new tool | 15:33 |
mgoddard | should we be using uwsgi instead then? | 15:33 |
yoctozepto | generally yes, except for glance which wants to stay with its old eventlet | 15:33 |
mgoddard | ok, we have two separate things here | 15:34 |
mnasiadka | yoctozepto: and we have a change for glance to use mod_wsgi | 15:34 |
generalfuzz | I got exceptions when I configured services with uwsgi and defined the certs. I can revisit that today to pinpoint the exceptions | 15:34 |
yoctozepto | mnasiadka: block it | 15:35 |
mgoddard | 1. backend tls - general pattern, usage, etc | 15:35 |
mgoddard | 2. backend web server | 15:35 |
mgoddard | if 2 is contentious we can continue with 1 | 15:35 |
yoctozepto | apache can do mod_proxy | 15:36 |
mgoddard | but let's not go adding mod_wsgi everywhere if its genuinely deprecated | 15:36 |
yoctozepto | generalfuzz: what broke with tls in uwsgi? | 15:36 |
mnasiadka | https://governance.openstack.org/tc/goals/selected/pike/deploy-api-in-wsgi.html#uwsgi-vs-mod-wsgi | 15:36 |
mnasiadka | (it's pike - but states devstack done move to uwsgi) | 15:36 |
mgoddard | I'm fairly sure OSA supports uwsgi | 15:36 |
yoctozepto | "with the intent that the mod_wsgi support is deleted from devstack in Queens." | 15:36 |
openstackgerrit | Merged openstack/kolla stable/train: CentOS 8: base and openstack-base images https://review.opendev.org/709537 | 15:36 |
mgoddard | #link https://docs.openstack.org/ansible-role-uwsgi/latest/ | 15:37 |
yoctozepto | yeah, osa is uwsgi | 15:37 |
yoctozepto | or at least to some degree | 15:37 |
mgoddard | not in devstack doesn't mean deprecated | 15:37 |
generalfuzz | yoctozepto: there were python openssl exceptions. I will revisit today | 15:38 |
mgoddard | generalfuzz: would be interesting to see what you changed to get uwsgi going | 15:39 |
yoctozepto | mgoddard: right but if osa and devstack push towards uwsgi, then mod_wsgi becomes obsolete and can break | 15:40 |
mgoddard | what about tripleo? | 15:40 |
yoctozepto | also see: https://bugs.launchpad.net/neutron/+bug/1864418 | 15:41 |
openstack | Launchpad bug 1864418 in neutron "has wrong with use apache to start neutron api in docker container" [Undecided,New] | 15:41 |
yoctozepto | this might be neutron behind mod_wsgi issue | 15:41 |
yoctozepto | mgoddard: good question | 15:41 |
openstackgerrit | Merged openstack/kolla stable/train: CentOS 8: Update packages in images https://review.opendev.org/709202 | 15:42 |
mgoddard | looks like quite a lot of wsgi in tripleo | 15:42 |
mgoddard | which means kolla images probably need to keep apache packages | 15:42 |
mgoddard | (unless we get them to override) | 15:42 |
generalfuzz | mgoddard: uwsgi is supported out of the box for nova. I'll need to look at Placement + keystone | 15:43 |
mgoddard | ok, sounds like more research required on wsgi vs. uwsgi | 15:43 |
yoctozepto | generalfuzz: keystone is uwsgi-only in osa | 15:43 |
mnasiadka | mgoddard: well, in theory we could support both | 15:43 |
yoctozepto | mhm, tripleo looks mod_wsgi-only | 15:43 |
generalfuzz | we may need a hybrid approach | 15:44 |
mgoddard | mnasiadka: no thanks :) | 15:44 |
yoctozepto | mgoddard: mnasiadka is right | 15:44 |
yoctozepto | stuffing both required parts in kolla is nobrainer | 15:44 |
mnasiadka | mgoddard: I think it will end up like this unfortunately | 15:44 |
mgoddard | why? | 15:44 |
yoctozepto | and we might want a phasing out approach | 15:44 |
yoctozepto | ^ | 15:44 |
yoctozepto | as it may break any day | 15:44 |
mnasiadka | for now it works I guess, so it's not critical ;) | 15:45 |
mgoddard | indeed | 15:45 |
yoctozepto | agreed | 15:45 |
mnasiadka | more convenient would be to have some common template or role that unifies mod_wsgi configs | 15:45 |
mgoddard | generalfuzz: I would suggest not adding any more mod_wsgi configs for now :) | 15:45 |
generalfuzz | I will re-look at uwsgi with TLS. | 15:46 |
yoctozepto | well, those two kinda contradict each other | 15:46 |
mgoddard | well no point in a common mod_wsgi role if we move to uwsgi | 15:46 |
*** Trevor_V has joined #openstack-kolla | 15:46 | |
mgoddard | screw wsgi, I need a whiskey | 15:46 |
generalfuzz | Is adding ability to execute the container with the "root" user an acceptable solution? | 15:47 |
mgoddard | let's try to get backend tls polished and merged with just keystone support | 15:47 |
yoctozepto | wsgiey | 15:47 |
generalfuzz | mgoddard: I will remove the nova + placement for now | 15:47 |
yoctozepto | it seriously should not be necessary | 15:47 |
mnasiadka | around uwsgi - I just hope uwsgi version between distro is at least a bit consistent, looking at mod_wsgi versions that we have now (and have to use medieval configs due to CentOS) | 15:48 |
generalfuzz | yoctozepto: how can I have the container run the apache script with sudo from k-a code? | 15:48 |
mgoddard | on the root user - normally we change it in the container image | 15:48 |
mgoddard | USER root | 15:49 |
yoctozepto | ^ | 15:49 |
mgoddard | question is whether this presents a transition problem for tripleo or k-a | 15:49 |
*** TrevorV has quit IRC | 15:49 | |
generalfuzz | so I should go into docker scripts in kolla as a related change? | 15:50 |
mgoddard | looks like tripleo might already use wsgi for now | 15:50 |
mgoddard | yes | 15:50 |
yoctozepto | or eventlet | 15:50 |
yoctozepto | hard to catch that | 15:50 |
mgoddard | I'd be interested to see what files the uwsgi config requires for TLS | 15:50 |
yoctozepto | as it's just running py script | 15:50 |
openstackgerrit | Merged openstack/kolla-ansible stable/train: CentOS 8: Support variable image tag suffix https://review.opendev.org/709534 | 15:50 |
mnasiadka | this looks like the easiest thing on earth: https://uwsgi-docs.readthedocs.io/en/latest/HTTPS.html | 15:51 |
mgoddard | i.e. if we merge backend tls support for wsgi, could we switch to uwsgi without any change in the user interface (config files) | 15:51 |
mnasiadka | So I'm pretty sure it doesn't work | 15:51 |
*** dasp has joined #openstack-kolla | 15:52 | |
yoctozepto | --https 0.0.0.0:8443,foobar.crt,foobar.key | 15:52 |
yoctozepto | well, that pretty much explains what tls really is | 15:52 |
generalfuzz | mgoddard: config files would change, since we would declare cert/key in .conf file | 15:53 |
hrw | mgoddard: on my server I use nginx to wrap uwsgi with tls | 15:53 |
mgoddard | generalfuzz: that's fine - we control those | 15:53 |
mgoddard | looks like it's just a key and cert, same as wsgi | 15:54 |
yoctozepto | we always needs this at min | 15:55 |
yoctozepto | and then any shim to convert/glue to the required form would be sufficient | 15:55 |
yoctozepto | like that ugly haproxy habing key with cert | 15:55 |
yoctozepto | who came up with that | 15:55 |
mgoddard | does devstack use mod_uwsgi or uwsgi binary? | 15:56 |
ktibi | mgoddard, kayobe can auto resize lvm thinpool on seed if the disk is increase ? | 15:56 |
mgoddard | ktibi: meeting time | 15:56 |
mgoddard | ok, seems this has taken most of the meeting | 15:57 |
mgoddard | #topic Ussuri release planning (kayobe) | 15:57 |
*** openstack changes topic to "Ussuri release planning (kayobe) (Meeting topic: kolla)" | 15:57 | |
yoctozepto | mgoddard: uwsgi | 15:57 |
mnasiadka | kayobe as wsgi app? | 15:57 |
yoctozepto | mgoddard: external binary | 15:57 |
mnasiadka | (just laughing to continue wsgi topic) | 15:58 |
yoctozepto | mnasiadka: yes, wsgify kayobe | 15:58 |
mgoddard | I don't have much to say other than I have more CentOS 8 patches for kayobe - reviews please dougsz & priteau :) | 15:58 |
mgoddard | uwsgi yoctozepto mnasiadka | 15:58 |
dougsz | :) | 15:58 |
priteau | will do | 15:58 |
mgoddard | thanks | 15:58 |
openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla stable/train: Bump train versions https://review.opendev.org/710067 | 15:59 |
* yoctozepto cannot be deployed as wsgi app under mnasiadka | 15:59 | |
mgoddard | I put together a testing checklist for kayobe & centos 8 | 15:59 |
mgoddard | https://etherpad.openstack.org/p/kolla-centos8 | 15:59 |
mgoddard | please add to it if you think of anything else | 15:59 |
*** TrevorV has joined #openstack-kolla | 15:59 | |
mgoddard | and if you have any time for testing please let me know | 16:00 |
mgoddard | (no doubt testing & fixing) | 16:00 |
mgoddard | we outta time | 16:00 |
mgoddard | thanks all | 16:00 |
*** diurnalist has joined #openstack-kolla | 16:00 | |
mgoddard | #endmeeting | 16:00 |
*** openstack changes topic to "Kolla IRC meetings on Wednesdays @ 15:00 UTC - see agenda @ https://goo.gl/OXB0DL | Whiteboard: https://etherpad.openstack.org/p/KollaWhiteBoard | IRC channel is *LOGGED* @ http://goo.gl/3mzZ7b" | 16:00 | |
openstack | Meeting ended Wed Feb 26 16:00:40 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:00 |
openstack | Minutes: http://eavesdrop.openstack.org/meetings/kolla/2020/kolla.2020-02-26-15.01.html | 16:00 |
openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/kolla/2020/kolla.2020-02-26-15.01.txt | 16:00 |
openstack | Log: http://eavesdrop.openstack.org/meetings/kolla/2020/kolla.2020-02-26-15.01.log.html | 16:00 |
yoctozepto | thanks mgoddard | 16:00 |
openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla stable/stein: Bump stein versions https://review.opendev.org/710068 | 16:01 |
hrw | for rocky we would need to do by hand checks as rocky-em is already on for some components | 16:02 |
*** Trevor_V has quit IRC | 16:03 | |
mgoddard | hrw: should we update the script to ignore that tag? | 16:03 |
mgoddard | ktibi: I have resized thinpools before, not sure about automatically by kayobe though | 16:04 |
ktibi | mgoddard, ok. I try to use --wipe-disks but kayobe don't remove lvm partition | 16:05 |
hrw | mgoddard: on all branches probably | 16:06 |
*** lile has quit IRC | 16:11 | |
priteau | ktibi: --wipe-disks should clear out all LVM info, have you checked if something on that disk is still mounted? | 16:12 |
hrw | mgoddard: https://paste.centos.org/view/26c75996 is quick change (rocky branch) | 16:12 |
ktibi | priteau, it's on seed VM and bifrost container is not created. | 16:13 |
*** lile has joined #openstack-kolla | 16:15 | |
hrw | mgoddard: master has -em ignoring. we just not cherrypicked it | 16:17 |
mgoddard | ah | 16:17 |
openstackgerrit | Merged openstack/kolla-ansible stable/train: CentOS 8: Deploy CentOS 8 containers https://review.opendev.org/709535 | 16:17 |
hrw | guess who wrote it... | 16:17 |
hrw | commit f528ad81026526af02610aa3e08200fd53b87ab9 | 16:17 |
hrw | Author: Mark Goddard <mark@stackhpc.com> | 16:17 |
hrw | Date: Mon May 20 19:00:33 2019 +0100 | 16:17 |
hrw | Ignore EM releases in version-check.py | 16:17 |
cosmicsound | i enable tls and define own tls in place, and in end i still get a self generate ssl how is this even possible? | 16:18 |
yoctozepto | spoiler alert was missed | 16:18 |
cosmicsound | or is this normal behaviour? | 16:18 |
openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla stable/stein: Ignore EM releases in version-check.py https://review.opendev.org/710073 | 16:18 |
yoctozepto | no, it's not | 16:18 |
yoctozepto | I run external tls and it works fine | 16:18 |
openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla stable/rocky: Ignore EM releases in version-check.py https://review.opendev.org/710074 | 16:18 |
cosmicsound | what vars you use | 16:18 |
cosmicsound | il share mine nnow | 16:18 |
cosmicsound | second | 16:18 |
cosmicsound | https://mdb.uhlhost.net/uploads/928d63db84b9ec23/image.png here are globals.yml | 16:19 |
hrw | marked Rocky one as RP+1 | 16:19 |
cosmicsound | haproxy-ca.crt haproxy-ca-internal.crt haproxy-internal.pem haproxy.pem have these in my /etc/kolla/certificates | 16:20 |
yoctozepto | cosmicsound | 16:21 |
yoctozepto | http://paste.openstack.org/show/790038/ | 16:21 |
yoctozepto | http://paste.openstack.org/show/790039/ | 16:22 |
openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: Fix fernet bootstrap and key distribution - follow up https://review.opendev.org/707080 | 16:22 |
openstackgerrit | Merged openstack/kolla-ansible stable/train: Add python3-dev[el] to bindep.txt https://review.opendev.org/709694 | 16:23 |
cosmicsound | yoctozepto , what was the command to get to the one file .pem | 16:35 |
cosmicsound | thats the normal crt with the .key into one .pem? | 16:36 |
yoctozepto | yeah, they are concatenated | 16:36 |
cosmicsound | -----END CERTIFICATE----- | 16:36 |
cosmicsound | -----BEGIN RSA PRIVATE KEY----- | 16:36 |
cosmicsound | i have first cert and then followed by key | 16:36 |
cosmicsound | inside nothing elese | 16:36 |
cosmicsound | il give it a go | 16:36 |
cosmicsound | so no eed for ca_chain] | 16:36 |
yoctozepto | I have cert, cert and key | 16:37 |
yoctozepto | whole chain to allow for match with root | 16:37 |
cosmicsound | not sure i get this | 16:38 |
cosmicsound | do you take your instance public? | 16:39 |
cosmicsound | or is just internal tls | 16:39 |
cosmicsound | root = ca authority who made the ssl? in my case is sectigo old comodo | 16:39 |
openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: OVN Support https://review.opendev.org/696841 | 16:42 |
yoctozepto | cosmicsound: certs are usually signed by an intermediate | 16:43 |
yoctozepto | cosmicsound: and only root is trusted | 16:43 |
yoctozepto | cosmicsound: so omitting intermediary from cert chain may render the connection untrusted | 16:43 |
hrw | mgoddard: found the reason for mistral-dashboard going backwards... | 16:43 |
hrw | mgoddard: will discuss with release team | 16:44 |
cosmicsound | yes so i need the provider trust chain | 16:44 |
cosmicsound | got it | 16:44 |
*** bengates has quit IRC | 16:50 | |
openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla stable/train: Bump train versions https://review.opendev.org/710067 | 16:53 |
cosmicsound | redeploying 🤞 | 17:05 |
cosmicsound | got it into one .pem all | 17:05 |
cosmicsound | ca chain cert and key | 17:05 |
cosmicsound | i am confused a bit with node_custom_config and node_config are they same? | 17:06 |
openstackgerrit | Merged openstack/kolla-ansible stable/train: CI: Move ansible installation & configuration to Ansible https://review.opendev.org/709717 | 17:06 |
cosmicsound | node_custom_config = /etc/kolla/config | 17:06 |
cosmicsound | certs have the {{ode_custom}}/certificates . do i need to put them in /config ? | 17:07 |
cosmicsound | *node | 17:07 |
ktibi | mgoddard, when bifrost start I can see in the container a process: git-remote-https origin https://opendev.org/openstack/ironic | 17:10 |
ktibi | the image need to be have internet access ? | 17:10 |
mgoddard | ktibi: unfortunately, yes | 17:14 |
ktibi | mgoddard, hum, any workarround ? | 17:15 |
mgoddard | ktibi: maybe you can pass some --skip-tags to bifrost? | 17:15 |
mgoddard | or add config to point to local repos | 17:15 |
ktibi | the playbook install bifrost in the container during the bootstrap ? | 17:16 |
mgoddard | ktibi: we run bifrost install when the container is created | 17:17 |
openstackgerrit | Merged openstack/kolla-ansible stable/train: CI: Use upper constraints when installing clients https://review.opendev.org/709751 | 17:17 |
openstackgerrit | Merged openstack/kolla-ansible stable/train: CI: Use python 3 for local kolla-ansible execution https://review.opendev.org/709718 | 17:17 |
mgoddard | then again with some --skip-tags during bootstrap | 17:17 |
mgoddard | maybe we are missing some tags to skip? | 17:17 |
ktibi | when you say "is created" it's during the build of the image ? | 17:18 |
mgoddard | yes | 17:19 |
openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/train: CentOS 8: Add deploy jobs in CI https://review.opendev.org/709536 | 17:19 |
ktibi | mgoddard, ok I can see a task in bifrost-prep, Download via GIT with a var bifrost_install_sources | 17:21 |
ktibi | how with kayobe can I override variables in this role ? | 17:21 |
cosmicsound | +1 yoctozepto | 17:21 |
cosmicsound | loving kolla more and more each time i pass one messy one like this | 17:21 |
cosmicsound | https://mdb.uhlhost.net/uploads/b09146c0dbe1a931/image.png | 17:21 |
cosmicsound | TLS works on public domain | 17:22 |
mgoddard | ktibi: https://docs.openstack.org/kayobe/latest/configuration/bifrost.html#custom-configuration | 17:22 |
cosmicsound | now need to solve internal domain naming | 17:22 |
ktibi | mgoddard, but I don't understand why I need to clone ironic repo. because it's a source image ? and not a binary ? | 17:22 |
hrw | bye | 17:23 |
mgoddard | ktibi: it's just part of bifrost installation | 17:24 |
ktibi | yes but the installation is during the build, not during the run no ? | 17:25 |
ktibi | like binary image | 17:25 |
ktibi | mgoddard, because I can see the call to the install playbook in the dockerfile, but why kayobe run again this playbook during the bootstrap of image :/ | 17:30 |
ktibi | mgoddard, ok I found, in kolla-ansible it's skip_package_install=true but in the playbook it's when: skip_install is not defined | 17:34 |
*** evrardjp has quit IRC | 17:34 | |
*** evrardjp has joined #openstack-kolla | 17:35 | |
ktibi | I try to add skip_install: true with your solution :) works | 17:40 |
*** gfidente is now known as gfidente|afk | 17:40 | |
* yoctozepto is glad, cosmicsound | 17:44 | |
* cosmicsound is gratefull for such a community | 17:45 | |
cosmicsound | The only one think I did not managed to get going | 17:46 |
cosmicsound | was allocation of ipv4 vip external | 17:46 |
cosmicsound | while if you remember on prechecks it used to pass the pinng | 17:46 |
cosmicsound | in the end no horizon was accessible | 17:46 |
cosmicsound | the bug i opened tried to explain this | 17:46 |
yoctozepto | cosmicsound: where are you showing the cert being deployed then? ;p | 17:47 |
cosmicsound | well this is made with haproxy off and keepalive | 17:47 |
*** ktibi has quit IRC | 17:48 | |
cosmicsound | and i used main public ip assigned to my bare servers | 17:48 |
cosmicsound | I have eno1 public and eno2 internal | 17:48 |
cosmicsound | 172.22.0.1/24 with .22 internal vip and eno1 main ipv4 as external ip yet not VIP since haproxy is off | 17:49 |
*** k_mouza has quit IRC | 17:49 | |
cosmicsound | with haproxy on it wont work, i added another address set it was same no ping so il debug this more | 17:50 |
cosmicsound | at first its netplan causing it | 17:50 |
cosmicsound | will try also bond approach since now its flat | 17:50 |
cosmicsound | bridge sorry | 17:50 |
cosmicsound | Now i made the main ssl work, will give it another chance | 17:51 |
cosmicsound | need more rtm on neutron | 17:52 |
cosmicsound | i know il make a terminal record session | 17:53 |
cosmicsound | is easiest to see what is done | 17:53 |
*** dougsz has quit IRC | 17:57 | |
*** diurnalist has quit IRC | 18:07 | |
*** lile has quit IRC | 18:11 | |
*** tonythomas has quit IRC | 18:21 | |
*** diurnalist has joined #openstack-kolla | 18:25 | |
*** lile has joined #openstack-kolla | 18:27 | |
*** priteau has quit IRC | 18:32 | |
*** igordc has joined #openstack-kolla | 18:37 | |
*** igordc has quit IRC | 18:43 | |
openstackgerrit | Merged openstack/kayobe master: CentOS 8: Use ansible_playbook_python for localhost dependencies https://review.opendev.org/708921 | 18:45 |
openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: OVN Support https://review.opendev.org/696841 | 18:48 |
openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: Fix fernet bootstrap and key distribution - follow up https://review.opendev.org/707080 | 18:49 |
osmanlicilegi | need a quick review for https://review.opendev.org/#/c/705608/ | 18:49 |
patchbot | patch 705608 - kolla-ansible - Fixes gnocchi-api script name for Ubuntu/Debian - 1 patch set | 18:49 |
mnasiadka | osmanlicilegi: done | 18:51 |
osmanlicilegi | thanks! | 18:51 |
openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: Add /run/netns bindmount to Neutron containers https://review.opendev.org/710051 | 18:54 |
cosmicsound | yoctozepto , this one is the answer on my precheks now | 19:39 |
cosmicsound | TASK [haproxy : Checking if kolla_internal_vip_address and kolla_external_vip_address are not pingable from any node] *********************************************** | 19:39 |
cosmicsound | ok: [compute-1] => (item={'address': '172.22.0.22', 'command': 'ping'}) | 19:39 |
cosmicsound | failed: [compute-1] (item={'address': '51.91.153.141', 'command': 'ping'}) => {"ansible_loop_var": "item", "changed": false, "cmd": ["ping", "-c", "3", "51.91.153.141"], "delta": "0:00:02.050088", "end": "2020-02-26 19:38:08.288743", "failed_when_result": true, "item": {"address": "51.91.153.141", "command": "ping"}, "rc": 0, "start": "2020-02-26 | 19:39 |
cosmicsound | 19:38:06.238655", "stderr": "", "stderr_lines": [], "stdout": "PING 51.91.153.141 (51.91.153.141) 56(84) bytes of data.\n64 bytes from 51.91.153.141: icmp_seq=1 ttl=64 time=0.022 ms\n64 bytes from 51.91.153.141: icmp_seq=2 ttl=64 time=0.077 ms\n64 bytes from 51.91.153.141: icmp_seq=3 ttl=64 time=0.084 ms\n\n--- 51.91.153.141 ping statistics ---\n3 | 19:39 |
cosmicsound | packets transmitted, 3 received, 0% packet loss, time 2048ms\nrtt min/avg/max/mdev = 0.022/0.061/0.084/0.027 ms", "stdout_lines": ["PING 51.91.153.141 (51.91.153.141) 56(84) bytes of data.", "64 bytes from 51.91.153.141: icmp_seq=1 ttl=64 time=0.022 ms", "64 bytes from 51.91.153.141: icmp_seq=2 ttl=64 time=0.077 ms", "64 bytes from 51.91.153.141: | 19:39 |
cosmicsound | icmp_seq=3 ttl=64 time=0.084 ms", "", "--- 51.91.153.141 ping statistics ---", "3 packets transmitted, 3 received, 0% packet loss, time 2048ms", "rtt min/avg/max/mdev = 0.022/0.061/0.084/0.027 ms"]} | 19:39 |
cosmicsound | alto the ip its pingable | 19:39 |
cosmicsound | its indeed not a floating ip, with a flowting ip this will pass just my dashboard will be gone | 19:39 |
yoctozepto | it must *not* be pingable *before* deployment | 19:39 |
yoctozepto | because it will be configured | 19:40 |
cosmicsound | right | 19:40 |
cosmicsound | because indeed this replies to pings | 19:40 |
cosmicsound | last log i showed you it passed just no console anywhere | 19:41 |
cosmicsound | will retry that range | 19:41 |
openstackgerrit | Dincer Celik proposed openstack/kolla-ansible stable/train: Use more permissive regex to remove the offending 127.0.1.1 https://review.opendev.org/710126 | 19:55 |
openstackgerrit | Dincer Celik proposed openstack/kolla-ansible stable/stein: Use more permissive regex to remove the offending 127.0.1.1 https://review.opendev.org/710127 | 19:55 |
yoctozepto | osmanlicilegi: no rocky? ^ | 20:02 |
*** diurnalist has quit IRC | 20:26 | |
*** kozhukalov has quit IRC | 20:33 | |
*** kozhukalov has joined #openstack-kolla | 20:36 | |
*** sean-k-mooney has quit IRC | 20:41 | |
*** diurnalist has joined #openstack-kolla | 20:41 | |
*** kplant has quit IRC | 20:49 | |
openstackgerrit | Merged openstack/kolla stable/stein: Ignore EM releases in version-check.py https://review.opendev.org/710073 | 21:09 |
openstackgerrit | Merged openstack/kolla stable/rocky: Ignore EM releases in version-check.py https://review.opendev.org/710074 | 21:09 |
openstackgerrit | Merged openstack/kolla-ansible master: Fixes gnocchi-api script name for Ubuntu/Debian https://review.opendev.org/705608 | 21:09 |
*** priteau has joined #openstack-kolla | 21:28 | |
*** kozhukalov has quit IRC | 21:29 | |
*** k_mouza has joined #openstack-kolla | 21:50 | |
*** k_mouza has quit IRC | 21:55 | |
*** priteau has quit IRC | 22:11 | |
*** JangwonLee has quit IRC | 22:31 | |
*** TrevorV has quit IRC | 22:38 | |
*** ktibi has joined #openstack-kolla | 23:04 | |
*** ktibi has quit IRC | 23:09 | |
*** JangwonLee has joined #openstack-kolla | 23:11 | |
*** cah_link1 has joined #openstack-kolla | 23:20 | |
*** cah_link has quit IRC | 23:23 | |
*** cah_link1 is now known as cah_link | 23:23 | |
*** JangwonLee has quit IRC | 23:45 | |
*** JangwonLee has joined #openstack-kolla | 23:47 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!