Tuesday, 2019-09-03

*** andrein has quit IRC00:31
*** gregoryo has joined #openstack-kolla00:47
*** andrein has joined #openstack-kolla01:02
*** radeks has quit IRC01:15
openstackgerritKien Nguyen proposed openstack/kolla master: Upgrade Haproxy to version 1.8.x  https://review.opendev.org/67117801:33
openstackgerritKien Nguyen proposed openstack/kolla-ansible master: Use specific Haproxy command  https://review.opendev.org/67118001:33
openstackgerritKien Nguyen proposed openstack/kolla-ansible master: Use specific Haproxy command  https://review.opendev.org/67118001:35
*** threestrands has joined #openstack-kolla01:37
*** hongbin has joined #openstack-kolla01:44
openstackgerritKien Nguyen proposed openstack/kolla-ansible master: Use specific Haproxy command  https://review.opendev.org/67118001:51
*** wuchunyang has joined #openstack-kolla02:26
*** andrein has quit IRC02:44
*** wuchunyang has quit IRC02:51
*** wuchunyang has joined #openstack-kolla02:53
*** wuchunyang has quit IRC02:58
*** skramaja has joined #openstack-kolla03:18
*** hongbin has quit IRC03:26
*** hongbin has joined #openstack-kolla03:28
*** gkadam has joined #openstack-kolla03:50
openstackgerritChason Chan proposed openstack/kolla-ansible master: Don't recursively modify file perms on start  https://review.opendev.org/67970004:10
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible stable/stein: Fix Zun Docker remote API host  https://review.opendev.org/67970104:26
*** hongbin has quit IRC04:29
*** hongbin has joined #openstack-kolla04:49
*** hongbin has quit IRC04:49
*** Luzi has joined #openstack-kolla04:59
*** dr_feelgood has joined #openstack-kolla05:18
*** ivve has joined #openstack-kolla05:35
*** wuchunyang has joined #openstack-kolla05:53
*** wuchunyang has quit IRC05:54
*** dr_feelgood has quit IRC05:55
*** dr_feelgood has joined #openstack-kolla06:04
*** dr_feelgood has quit IRC06:05
*** dr_feelgood has joined #openstack-kolla06:06
*** hamdyk has joined #openstack-kolla06:17
*** radeks has joined #openstack-kolla06:31
*** cah_link has joined #openstack-kolla06:44
openstackgerritChason Chan proposed openstack/kolla-ansible master: Don't recursively modify file perms on start  https://review.opendev.org/67970006:57
*** pcaruana has joined #openstack-kolla07:06
*** gfidente has joined #openstack-kolla07:10
hrwmorning07:14
*** threestrands has quit IRC07:18
mnasiadkamorning07:27
openstackgerritChason Chan proposed openstack/kolla-ansible master: Don't recursively modify file perms on start  https://review.opendev.org/67970007:35
hrw07:37
*** jbadiapa has quit IRC07:40
*** luksky has joined #openstack-kolla07:40
*** nathharp has joined #openstack-kolla07:42
*** priteau has joined #openstack-kolla07:46
hrwmnasiadka: did you started work on populating support matrix from unbuildable?07:50
mnasiadkahrw: not yet, had some $work to do ;)07:51
hrw;)07:51
*** jbadiapa has joined #openstack-kolla07:56
*** egonzalez has joined #openstack-kolla07:58
mnasiadkabetter somebody tell me why curl trunk.rdoproject.org has some problems in our gates :)08:01
hrw;D08:02
mnasiadkaI've seen it on ova and rax08:02
mnasiadkaovh08:02
hrwboth are trouble makers iirc08:02
*** dougsz has joined #openstack-kolla08:05
hrw1684 images on laptop...08:13
egonzalezmnasiadka, we change rdoproject with infra mirrors url08:17
*** k_mouza has joined #openstack-kolla08:18
egonzalezand there is an email with this topic [rdo-dev] [infra][outage] trunk.rdoproject.org down08:18
*** andrein has joined #openstack-kolla08:19
mgoddardmorning08:56
*** wuchunyang has joined #openstack-kolla08:58
openstackgerritOndrej Duchon proposed openstack/kolla-ansible master: Corrected notifications for auditing.  https://review.opendev.org/67854409:09
*** gregoryo has quit IRC09:10
hrwhttps://www.youtube.com/watch?v=LChMOMF8GPw - "My first 8K" intro for Atari 2600 VCS.09:11
hrwsomething different for machine from different era09:11
openstackgerritMerged openstack/kolla stable/rocky: Delete default gnocchi & aodh wsgi conf in Ubuntu binary  https://review.opendev.org/67961709:25
*** dr_feelgood has quit IRC09:32
*** k_mouza has quit IRC09:50
mnasiadkaegonzalez: for fetching the repo file? I don't think so :)09:53
*** wuchunyang has quit IRC09:54
*** dr_feelgood has joined #openstack-kolla09:57
*** ondrejduchon has joined #openstack-kolla09:58
*** ondrejduchon has quit IRC09:58
openstackgerritMark Goddard proposed x/kayobe master: Make SNAT iptables rules persistent on the seed  https://review.opendev.org/67973909:59
*** ianw has quit IRC10:00
*** k_mouza has joined #openstack-kolla10:03
openstackgerritMark Goddard proposed x/kayobe master: Make SNAT iptables rules persistent on the seed  https://review.opendev.org/67973910:04
*** ianw has joined #openstack-kolla10:08
*** k_mouza has quit IRC10:14
*** k_mouza has joined #openstack-kolla10:15
openstackgerritMerged openstack/kolla stable/rocky: Bump service versions for Rocky  https://review.opendev.org/67944310:15
hrwhttps://marcin.juszkiewicz.com.pl/2019/09/03/my-first-8k-intro/10:19
*** egonzalez has quit IRC10:22
openstackgerritMerged openstack/kolla stable/stein: Bump service versions for Stein  https://review.opendev.org/67944410:22
*** ianw has quit IRC10:27
*** ianw has joined #openstack-kolla10:28
*** ianw has quit IRC10:28
*** ianw has joined #openstack-kolla10:29
*** ianw has quit IRC10:34
*** ianw has joined #openstack-kolla10:36
mnasiadkahrw: nice code :D10:37
*** ianw_ has joined #openstack-kolla10:41
*** ianw_ has quit IRC10:41
*** ianw_ has joined #openstack-kolla10:43
*** ianw_ has quit IRC10:44
*** ianw has quit IRC10:44
*** ianw has joined #openstack-kolla10:44
*** ianw has quit IRC10:46
*** ianw has joined #openstack-kolla10:50
*** ianw has quit IRC10:52
*** ianw has joined #openstack-kolla10:52
*** andrein is now known as andrein_10:54
openstackgerritDmitry Tantsur proposed openstack/kolla master: ironic-inspector: install the dnsmasq subpackage explicitly  https://review.opendev.org/67975111:05
*** luksky has quit IRC11:08
openstackgerritMerged openstack/kolla master: disable 'hacluster-pcs' on aarch64 due to lack of package  https://review.opendev.org/67891011:12
*** CosmicSound has quit IRC11:14
hrwmnasiadka: thx11:15
*** CosmicSound has joined #openstack-kolla11:18
*** andrein_ has quit IRC11:21
*** kplant has joined #openstack-kolla11:28
watersjsfsdf11:44
*** wuchunyang has joined #openstack-kolla11:56
watersjMariadb/garlea, for unexpected outage is it supposed to failover? For a planned outage, where is stop mariadb container properly everything works. But if there is a power cord pulled, mariadb becomes unresponsive (2 controller nodes with HA configured).11:56
mgoddardwatersj: have you tried using kolla-ansible mariadb_recovery?11:57
watersjyes, but thats to recovery. I was hoping for zero impact on cluster while still having 1 node up and operational11:58
watersjnot all events are planned, and I was under impression mariadb/galera would stay up and functional.12:00
mgoddardwatersj: with 2 nodes you lose quorum when one goes down. Try adding a third12:01
*** andrein has joined #openstack-kolla12:02
watersjyes, that was what I was wondering if I needed an additional controller node12:02
watersjso 3 is required12:02
watersj?12:02
mgoddardyes12:04
watersjty12:04
*** wuchunyang has quit IRC12:04
andreinHello everyone, I'm trying to deploy magnum using kolla-ansible (stein) and I've hit the following error when I try to deploy it: FAILED! => {"changed": false, "msg": "User magnum_trustee_domain_admin is not valid"}12:06
andreinunder TASK [magnum : Creating Magnum trustee user role]12:06
andreinany ideas what could cause this? running the deploy again fails on the previous task (Creating Magnum trustee user) with FAILED! => {"changed": false, "extra_data": null, "msg": "Error in creating user magnum_trustee_domain_admin. (409) Client Error for url: http://172.16.0.250:35357/v3/users Conflict occurred attempting to store user - Duplicate entry found with name magnum_trustee_domain_admin at domain ID12:07
andreinb929576d45b746319a208ff170f1d466."}12:07
hrwut12:10
* hrw out12:10
mgoddardandrein: what region name are you using?12:17
andreinmgoddard: I'm using the default RegionOne12:17
andreinI think I'm hitting this bug: https://bugs.launchpad.net/kolla-ansible/+bug/171401112:17
openstackLaunchpad bug 1714011 in kolla-ansible "Reconfigure failing on creating magnum_trustee_domain_admin when domain_specific_drivers_enabled = True is set in keystone.conf" [Undecided,Incomplete] - Assigned to yuqian (roger-yu)12:17
mgoddardor do you have a non-default domain?12:17
andreinyes, we've also enabled a LDAP domain.12:17
andreinwill try the workaround in the latest comment (pulling kolla-toolbox from master)12:18
*** k_mouza_ has joined #openstack-kolla12:19
mgoddardandrein: ok. If you find a solution please share12:19
mgoddardandrein: bear in mind we don't test containers across different releases, so you might be better off building a custom stein image of kolla-toolbox with an updated ansible12:20
mgoddardandrein: or cherry-picking that fix (https://opendev.org/openstack/kolla/commit/e3108d93c69dbad0c575da719f842d36c195a673)12:20
*** k_mouza has quit IRC12:22
andreinmgoddard: using kolla-toolbox from master fixes the issue as far as I can tell12:22
mgoddardok. It might not be a good long term solution because the contents of the master image may change, but you could take a snapshot12:22
*** nathharp has quit IRC12:23
*** watersj has quit IRC12:26
*** jistr has quit IRC12:29
*** jistr has joined #openstack-kolla12:29
*** jistr has quit IRC12:30
andreinmgoddard: agreed, I'm hoping that change will be cherry picked into stein though12:31
mgoddardandrein: I'm just discussing this with mnasiadka. I think the change is too big to backport as it is12:32
mgoddardansible upgrade from 2.2 to 2.8 is a lot for a stable branch12:32
mnasiadkacloudnull: https://review.opendev.org/#/c/675614/ - if you would have time to look into why TripleO job is failing - I would be grateful12:32
*** weshay has joined #openstack-kolla12:34
*** gkadam has quit IRC12:37
*** jistr has joined #openstack-kolla12:40
*** redrobot has quit IRC12:55
*** pcaruana has quit IRC13:03
*** wuchunyang has joined #openstack-kolla13:06
*** k_mouza has joined #openstack-kolla13:07
*** k_mouza_ has quit IRC13:11
*** signed8b_ has quit IRC13:13
*** signed8bit has joined #openstack-kolla13:13
*** luksky has joined #openstack-kolla13:16
*** wuchunyang has quit IRC13:21
*** wuchunyang has joined #openstack-kolla13:21
*** pcaruana has joined #openstack-kolla13:34
*** k_mouza_ has joined #openstack-kolla13:35
*** redrobot_ has joined #openstack-kolla13:36
*** redrobot_ is now known as redrobot13:37
*** k_mouza has quit IRC13:38
*** Luzi has quit IRC13:41
cloudnullmnasiadka looking now13:42
*** KeithMnemonic has joined #openstack-kolla13:44
*** watersj has joined #openstack-kolla13:45
*** pcaruana has quit IRC13:47
*** cah_link has quit IRC13:49
*** pcaruana has joined #openstack-kolla13:57
*** wuchunyang has quit IRC14:00
*** wuchunyang has joined #openstack-kolla14:02
*** k_mouza has joined #openstack-kolla14:02
henriqueof1Good morning y'all, I see that there are alway code changes to GitHub repos and the images are published under the same tag, is it okay to delete and pull again the container to get latest code changes?14:03
*** k_mouza_ has quit IRC14:05
mgoddardhenriqueof1: that should be fine. No need to delete, just pull14:06
*** pcaruana has quit IRC14:07
henriqueof1mgoddard: Okay, thanks!14:07
henriqueof1I see some significant changes to magnum project lately and want to test.14:07
*** pcaruana has joined #openstack-kolla14:10
openstackgerritMerged openstack/kolla-ansible stable/rocky: Use net_default_mac in ansible/roles/ironic/templates/ironic_pxe_uefi.default.j2  https://review.opendev.org/67963914:18
*** wuchunyang has quit IRC14:20
openstackgerritMerged openstack/kolla-ansible stable/rocky: Use secure websocket for nova serial console proxy when TLS enabled  https://review.opendev.org/67963714:23
*** wuchunyang has joined #openstack-kolla14:24
*** hamdyk has quit IRC14:40
*** wuchunyang has quit IRC15:04
*** wuchunyang has joined #openstack-kolla15:05
*** wuchunyang has quit IRC15:09
*** skramaja has quit IRC15:10
*** wuchunyang has joined #openstack-kolla15:10
*** luksky has quit IRC15:13
openstackgerritMerged openstack/kolla-ansible master: Add support to enable l3 port-forwarding plugin  https://review.opendev.org/67967815:16
hrwmnasiadka: is http://paste.debian.net/1098576/ proper way?15:18
openstackgerritMerged openstack/kolla-ansible master: Remove Heat environment file disabling deprecated plugins  https://review.opendev.org/67486715:23
*** wuchunyang has quit IRC15:24
*** pcaruana has quit IRC15:27
*** wuchunyang has joined #openstack-kolla15:28
hrwmnasiadka: https://paste.fedoraproject.org/paste/7-PCVfhyIK9KupYALUselQ15:28
*** wuchunyang has quit IRC15:37
*** mixor has joined #openstack-kolla15:38
watersjll15:39
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Set any_errors_fatal for all plays  https://review.opendev.org/66687615:40
openstackgerritMark Goddard proposed openstack/kolla-ansible master: DNM: Test any_errors_fatal  https://review.opendev.org/67981115:40
openstackgerritOndrej Duchon proposed openstack/kolla-ansible master: Added support for custom logging.conf  https://review.opendev.org/67981215:40
openstackgerritMerged x/kayobe master: Use earliest-version 6.0.0 for Stein reno  https://review.opendev.org/67945315:40
*** wuchunyang has joined #openstack-kolla15:41
watersjadding an additional control node fails with mariadb container doesn't exist. Is there a way to create container first using kolla-ansible? (and maria_recovery expects container to exist to run)15:41
*** cah_link has joined #openstack-kolla16:03
mgoddardwatersj: are you using stable/stein or 8.0.0?16:03
mgoddardthere was a bug in 8.0.0, try stable/stein16:03
watersji am using stable/stein16:04
watersjor atleast I think i am, openstack_release: "stein"16:05
watersjshould I change my global to "stable/stein"16:05
*** cah_link has quit IRC16:07
*** cah_link has joined #openstack-kolla16:07
*** wuchunyang has quit IRC16:07
watersjmgoddard, guessing you mean from git repo, not the pip install16:09
watersjcloning now16:10
*** BjoernT has joined #openstack-kolla16:14
*** tonythomas has joined #openstack-kolla16:25
openstackgerritMark Goddard proposed openstack/kolla-ansible master: DNM: Test any_errors_fatal  https://review.opendev.org/67981116:25
mgoddardwatersj: correct16:26
mgoddardkolla-ansible git repo, stable/stein16:26
*** mixor has quit IRC16:26
*** jonaspaulo has joined #openstack-kolla16:37
openstackgerritDoug Szumski proposed openstack/kolla-ansible master: Fix Nova cell search  https://review.opendev.org/67982316:47
*** k_mouza_ has joined #openstack-kolla16:49
*** k_mouza_ has quit IRC16:51
*** k_mouza has quit IRC16:52
openstackgerritDoug Szumski proposed openstack/kolla-ansible master: Fix Nova cell search  https://review.opendev.org/67982317:13
*** pcaruana has joined #openstack-kolla17:29
*** dougsz has quit IRC17:36
*** priteau has quit IRC17:43
*** luksky has joined #openstack-kolla17:52
openstackgerritMerged openstack/kolla stable/queens: Bump service versions for Queens  https://review.opendev.org/67944218:18
*** gfidente is now known as gfidente|afk18:43
mnasiadkahrw: good job! Push it to the review :)18:43
openstackgerritMerged x/kayobe master: Include branch name in URL for IPA on master  https://review.opendev.org/67800918:50
openstackgerritMerged x/kayobe master: Revert "Use OpenStack Stein release"  https://review.opendev.org/67694118:50
*** igordc has joined #openstack-kolla19:02
*** cally725 has joined #openstack-kolla19:13
cally725Anybody tried to set mtu 9000 in kola installation ?19:13
watersjcally725, not I, are you wanting jumbo frames for your storage? might want to dedicate a network card(s) from storage19:22
cally725watersj that would be more for inter vm communication19:23
cally725havein br-int and br-tun mtu set to 900019:24
watersjcloning now19:29
kplantcally725: yes i use 9k bytes19:31
kplantglobal_physnet_mtu = 9000 in [DEFAULT] on neutron.conf19:31
kplantand path_mtu = 9000 in [ml2] of ml2_conf.ini19:31
cally725kplant cool how did you deploy you special config ? I set path_mtu = 9000 in ml2_conf.ini and global_physnet_mtu = 9000 in neutron.conf19:32
kplanti specified them in /etc/kolla/config19:33
kplantthe config override path19:33
cally725kplant I see that after deployment the values are distributed in the config files but the br-int, br-ex and br-tun are still set to mtu 150019:34
cally725kplant yes this is what I used /etc/kola/config19:34
cally725kplant I am on rocky19:34
kplantyou can ignore the mtus on the ovs bridges19:34
kplantwell not br-ex19:35
cally725kplant Ha ok !!19:35
kplantdid you also configure your NICs for 9k?19:35
cally725yes19:35
cally725I have mtu 9000 set on all interfaces19:36
cally725I see it when doing ip a19:36
cally725sorry I am running newton not rocky19:37
kplantand br-ex is 1500?19:39
cally725yes 150019:39
cally725and the vm tap are set to mtu 145019:39
cally7257: qbr3508625b-a8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 100019:39
kplanthmm19:40
kplantdid you set those values to 9000 before or after deploying19:41
cally725yes I  destroyed everiting , set the values, and redeployed19:41
kplantwhat's the exact paths of the files you set them in19:42
cally725I put them in /etc/kola/config and in /etc/kola/config/neutron to be sure19:43
cally725I see that on my compoute node in the /etc/kola/neutron-l3-agent/neutron.conf that the global_physnet_mtu = 9000 is set19:44
cally725same in compute node /etc/kola/neutron-openvswitch-agent19:45
cally725let me check in the docker itself...19:45
kplantdid you check if those are the right options for newton?19:46
kplantnewtn's pretty old :D19:46
cally725that's a thing I did not find the setting specific to a release19:46
cally725I have seen to set those value in a generic kola document19:47
cally725which version do you use ?19:48
kplantstein19:49
kplantreally all i do is set my NICs to 9k, by way of jinja, then those two values and deploy19:49
cally725which os do you use ?19:50
kplantcentos 719:50
cally725I have ubuntu 160419:50
cally725ok you see br-ex set to 9000 in your system ?19:51
kplant895019:51
kplant-50 for vxlan overhead19:51
cally725ok yep19:52
kplanterm i lied19:52
cally725and the vms do they get mtu 8950 as well ?19:52
kplantbr-ex is 900019:52
kplantbr-int is 895019:52
kplantdepending on what node you're looking at19:52
kplantyes, they do19:52
kplantqvbd4376f76-62@qvod4376f76-62: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8950 qdisc noqueue master qbrd4376f76-62 state UP group default qlen 100019:52
cally725ok pfffff….. maybe my version then19:52
cally725wow cool that's what I need19:53
kplantquitting time19:53
kplanthave a good one19:53
*** kplant has quit IRC19:53
cally725thanks19:53
cally725at least I know that it is possible now and that I am doing the good thing I will try with a newer release19:54
cally725exit19:56
watersjloss connectivity to an instance after "primary" mariadb goes offline. Have 3 controller nodes, if I loss any 1 of other 2 nodes the instance have connectivity (stable/stein)19:59
*** pcaruana has quit IRC20:00
*** wuchunyang has joined #openstack-kolla20:06
*** wuchunyang has quit IRC20:10
osmanlicilegiwatersj: are you having that issue on ubuntu?20:17
watersjosmanlicilegi, no centos (centos hosts, centos images)20:19
osmanlicilegiI had a similar issue with stein, but on ubuntu. Fixed it by upgrading haproxy from 1.8 to 2.0.20:20
watersjThink I missed adding kolla_internal_fqdn: <myos.fqdn>, I'll look at haproxy too, ty20:21
osmanlicilegiCheck the galera cluster health by connecting each nodes mariadb directly. If they're ok, try checking over haproxy. If you hit the issue over haproxy, you need to upgrade.20:24
*** radeks has quit IRC20:37
watersjI checked marias and they all said sync'd20:43
watersjwow, maybe I should change containers, haproxy-1.5.18-8.el7.x86_6420:44
*** cah_link has quit IRC20:52
osmanlicilegi1.5 is too old for haproxy20:56
*** dougsz has joined #openstack-kolla20:57
watersjlooking like my deploy method is in error, how are most people doing kollla-ansible installs? binary or source? thel or debian derivatives ?20:58
watersjthel = rhel20:59
osmanlicilegiI go with ubuntu binary with private docker repo20:59
osmanlicilegiPrivate repo lets you free if you need to customize images21:00
watersjand what about your docker host?21:00
osmanlicilegiubuntu bionic everywhere :)21:01
osmanlicilegibtw, https://www.softwarecollections.org/en/scls/rhscl/rh-haproxy18/ can help you21:03
*** dougsz has quit IRC21:05
osmanlicilegiwatersj: this is my template-overrides.j2 file for haproxy upgrade: https://paste.ee/p/6auHx21:12
watersjosmanlicilegi, ty for the info.21:12
*** dougsz has joined #openstack-kolla21:23
*** luksky has quit IRC21:27
*** dougsz has quit IRC21:48
*** BjoernT has quit IRC22:33
*** mgoddard has quit IRC22:47
*** mgoddard has joined #openstack-kolla22:47
openstackgerritManuel Rodriguez proposed openstack/kolla-ansible stable/stein: Add support to enable l3 port-forwarding plugin  https://review.opendev.org/67987623:01
*** jonaspaulo has quit IRC23:04
*** tonythomas has quit IRC23:14
*** threestrands has joined #openstack-kolla23:26

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!