Wednesday, 2019-08-28

*** keper7 has quit IRC00:55
*** k_mouza has joined #openstack-kolla01:33
*** spsurya has joined #openstack-kolla02:11
*** dpawlik has joined #openstack-kolla02:12
*** zaitcev has joined #openstack-kolla02:14
*** k_mouza has quit IRC02:16
*** dpawlik has quit IRC02:16
zaitcevI don't know a thing about Docker and I am working on a bug02:20
zaitcevWe have a dockerized rsyncd, apparently coming from this: docker/swift/swift-rsyncd/Dockerfile.j202:21
zaitcevthe problem is, the container cannot restart because it leaves behind a file that it itself cannot overwrite when it restarts:02:22
zaitcev /var/lib/docker/overlay2/fdfbd45c034ecf9cf3f91722862386c805ecec3126d92dc7631a0437d62a6db1/diff/run/rsyncd.pid02:22
zaitcevI started question just how it manages to leave it, because /run is supposed to be tmpfs mounted and /var/run symlinks to it.02:23
zaitcevErgo, we forgot to mount a fresh /run in this container02:23
zaitcevBut looking at Kolla's source for the container, I don't know where to begin... Like, how the mounts (or bind mounts) are regulated inside a docket container.02:24
*** keper7 has joined #openstack-kolla02:29
*** itlinux has joined #openstack-kolla02:44
*** itlinux has quit IRC02:45
*** keper7 has quit IRC02:51
*** gkadam has joined #openstack-kolla03:36
*** gkadam has quit IRC03:36
*** zaitcev has quit IRC04:06
*** dpawlik has joined #openstack-kolla04:13
*** k_mouza has joined #openstack-kolla04:16
*** dpawlik has quit IRC04:17
*** zaitcev has joined #openstack-kolla04:20
*** k_mouza has quit IRC04:20
*** itlinux has joined #openstack-kolla04:39
*** dave-mccowan has quit IRC04:40
*** itlinux has quit IRC04:52
*** dr_feelgood has joined #openstack-kolla05:05
*** dr_feelgood has quit IRC05:09
*** itlinux has joined #openstack-kolla05:10
*** itlinux has quit IRC05:19
*** dr_feelgood has joined #openstack-kolla05:38
*** shyamb has joined #openstack-kolla05:39
*** dr_feelgood has quit IRC05:43
*** shyamb has quit IRC05:44
*** Luzi has joined #openstack-kolla05:46
*** skramaja has joined #openstack-kolla06:13
yoctozeptozaitcev: hi, what's the exact error message you get06:14
*** dpawlik has joined #openstack-kolla06:15
*** hamdyk has joined #openstack-kolla06:17
yoctozeptoand also when exactly it happens, and what is the used release of kolla / kolla-ansible06:20
*** shyamb has joined #openstack-kolla06:20
yoctozeptomorning channel06:32
*** skramaja has quit IRC06:50
*** skramaja has joined #openstack-kolla06:54
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637607:09
mnasiadkamorning07:09
yoctozeptomnasiadka: https://review.opendev.org/678276 ;-)07:15
mnasiadkadone07:25
mnasiadkayoctozepto: https://review.opendev.org/#/c/675614/ :)07:25
yoctozeptomnasiadka: just did, you late07:26
mnasiadkagood07:26
yoctozeptofinal touches07:26
yoctozeptolet's make fluentd great again07:26
yoctozepto;D07:26
yoctozeptoalso, why did we include td.repo if we were not using td on rhel? did you check that?07:27
*** ivve has joined #openstack-kolla07:28
mnasiadkayoctozepto: somebody removing td-agent on rhel-derivatives in the past did not remove td.repo file07:32
mnasiadkabloody sudoers, I'll rename them to td-agent-sudoers and fluentd-sudoers :D07:33
mnasiadkabecause now the naming sucks07:33
yoctozeptoindeed, then adapt template07:33
yoctozeptogreat job anyways07:34
mnasiadkawell, I'll merge it into one sudoers file, it's mixed up already...07:35
mnasiadkaand I don't get why somebody did this in one line:07:35
mnasiadka%kolla ALL=(root) NOPASSWD: /bin/chown fluentd\:kolla /var/log/kolla, /usr/bin/chown fluentd\:kolla /var/log/kolla07:36
mnasiadkatwo times the same command?07:36
mnasiadkaah, different paths07:36
* mnasiadka blind in the morning07:36
yoctozeptono problem, mornings are worst :-)07:41
yoctozeptowe could certainly merge it into one file07:42
yoctozeptotd-agent and fluentd being in one place could not hurt07:42
yoctozeptothen it's a matter of line removal when fluentd is completely gone07:42
*** shyamb has quit IRC07:42
yoctozeptowell, two lines to be precise07:42
openstackgerritRadosław Piliszek proposed openstack/kolla master: Restore TripleO CI as voting and gated  https://review.opendev.org/67722807:43
mnasiadkayoctozepto: well, we use fluentd on Debian, I think hrw did some changes recently07:44
mnasiadkayoctozepto: around running this line https://opendev.org/openstack/kolla/src/branch/master/docker/fluentd/Dockerfile.j2#L70 only on Debian - I tend to disagree - seems Ubuntu !x86_64 also runs this07:45
*** egon^ has left #openstack-kolla07:47
yoctozeptomnasiadka: yeah, sorry, this is my morning blinddness for you :-)07:48
mnasiadkahaha :)07:48
mnasiadkagood that I'm not the only one retarded in the morning :)07:48
openstackgerritMerged openstack/kolla-ansible master: Add support for using custom Logstash patterns  https://review.opendev.org/67488407:48
openstackgerritMerged openstack/kolla-ansible master: Support configuration of trusted CA certificate file  https://review.opendev.org/67670907:48
openstackgerritMerged openstack/kolla-ansible master: Fixes Monasca log transformer UTC offset exception  https://review.opendev.org/67566007:48
*** zaitcev has quit IRC07:50
yoctozeptomnasiadka: I get 'retarded' again in the evening though, brain muscles are getting tired then07:50
yoctozeptocola drinks help a little bit, tons of sugar dissolved in black water :D07:52
*** zaitcev has joined #openstack-kolla08:02
mnasiadkayoctozepto: well, I'm starting to be lost in fluentd :D08:04
mgoddardmorning koalas08:05
yoctozeptomorning mgoddard08:05
*** zaitcev_ has joined #openstack-kolla08:06
mnasiadkayoctozepto: centos x86_64 and ubuntu x86_64 use td-agent and 0.14, debian (all archs?) and ubuntu non x86_64 uses fluentd and 0.14, and centos non x86_64 uses fluentd and 0.12 - how can I not loose sanity in all this?08:06
yoctozeptomnasiadka: by using proper variables in proper conditional branches08:06
yoctozeptoyou already did a great service to readability08:06
hrwmorning08:08
*** dougsz has joined #openstack-kolla08:08
mnasiadkayoctozepto: I know what should be the end goal, I'm rather complaining about the process of getting there :)08:09
*** zaitcev has quit IRC08:09
hrwmnasiadka: sep/oct td-agent 4 will be released. should cover everything we care and then ppc64le will suxx08:10
mnasiadkahrw: so it's an endless game with fluentd :)08:10
openstackgerritMerged openstack/kolla-ansible master: Moves monasca-thresh java.io.tmpdir to existing docker volume  https://review.opendev.org/67477508:13
openstackgerritMark Goddard proposed x/kayobe stable/stein: Add support for Arista switches  https://review.opendev.org/67900908:23
BlinkizDoes Kolla-ansible work with Python 3.7.4 on stein (8.0.0)?08:24
openstackgerritMichal Nasiadka proposed openstack/kolla master: Change fluentd to td-agent on CentOS  https://review.opendev.org/67561408:24
yoctozeptoBlinkiz: yup, it should08:25
BlinkizI try it then08:25
*** zaitcev_ has quit IRC08:30
yoctozeptomnasiadka: I have no idea how you manage to decrease complexity in one place, just to increase it in another :-)08:32
mnasiadkayoctozepto: Sometimes you need to increase complexity first, and then decrease it :)08:33
*** shyamb has joined #openstack-kolla08:33
yoctozeptomnasiadka: you just want me to constantly review :D08:36
mnasiadkayoctozepto: naah, now I wanted Zuul to tell me what breaks :)08:37
yoctozeptomnasiadka: k, take your time, break it and stick it back08:37
openstackgerritMichal Nasiadka proposed openstack/kolla master: Change fluentd to td-agent on CentOS  https://review.opendev.org/67561408:40
yoctozeptoweird error in ubuntu k-a08:41
yoctozeptohttps://80761cae90bb201ddf2a-de5ae15935168409da4576fce7897429.ssl.cf2.rackcdn.com/677228/3/check/kolla-ansible-ubuntu-source/309239f/primary/logs/ansible/deploy08:41
yoctozepto"oslo_db.exception.DBError: (pymysql.err.InternalError) (1832, \"Cannot change column 'network_id': used in a foreign key constraint 'subnets_ibfk_1'\")",08:41
yoctozepto        "[SQL: ALTER TABLE subnets MODIFY network_id VARCHAR(36) NOT NULL]",08:41
yoctozeptoRunning Neutron bootstrap container08:41
mnasiadkasaw that once, but never again08:42
*** zaitcev_ has joined #openstack-kolla08:43
openstackgerritMerged openstack/kolla-ansible stable/queens: Constrain the size of Docker logs  https://review.opendev.org/67886908:43
openstackgerritMark Goddard proposed openstack/kolla-ansible stable/stein: Moves monasca-thresh java.io.tmpdir to existing docker volume  https://review.opendev.org/67901208:48
openstackgerritMark Goddard proposed openstack/kolla-ansible stable/rocky: Moves monasca-thresh java.io.tmpdir to existing docker volume  https://review.opendev.org/67901308:48
yoctozeptomnasiadka, mgoddard, hrw: kolla(-ansible) on-cloud ipv6 officially broken on centos 7, ovs affected the same :<08:54
mgoddardyoctozepto: :(08:55
mgoddardis tripleo adding ipv6 support?08:56
yoctozeptohttps://bugzilla.redhat.com/show_bug.cgi?id=174612308:56
openstackbugzilla.redhat.com bug 1746123 in libvirt "Isolated IPv6 bridge network does not work with br_netfilter being on" [Unspecified,New] - Assigned to libvirt-maint08:56
yoctozeptomgoddard: no idea08:56
yoctozeptocloudnull: ^ ?08:56
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637608:57
yoctozeptowell, it could be the latest 7.608:58
yoctozeptono 7.7 ready and I have no older servers around08:58
yoctozeptoargh08:58
mgoddardyoctozepto: does this affect control plane, data plane or both?08:59
yoctozeptomgoddard: I meant on-cloud = data plane08:59
mgoddardok08:59
yoctozeptoI discovered it by wiring up virtualized control plane for our ipv6 support09:00
yoctozeptobaremetal should work fine still09:00
yoctozeptothis applies to libvirt bridges09:00
yoctozeptoor more precisely to ip6tables being lame ;p09:00
yoctozeptomgoddard: I posted on #tripleo, it might be more clear if you read there and the bug report I made09:02
yoctozeptoall right, let's continue the ipv6 control plane battle after this nasty disturbance ;(09:03
mgoddardyoctozepto: have you tried looking at net.bridge.bridge-nf-call-ip6tables?09:04
yoctozeptomgoddard: yeah, one can disable it and then it all works but then ip6tables is never consulted for this traffic kind09:05
yoctozeptoso no per-vm filtering09:05
mgoddardright. might be worth adding to the bug09:06
yoctozeptomgoddard: yeah, I could have been more precise09:07
yoctozepto;p09:07
yoctozeptohttp://codesearch.openstack.org/?q=call-ip6tables&i=nope&files=&repos=09:07
yoctozeptothis is interesting09:07
yoctozeptoesp.09:07
yoctozeptohttps://opendev.org/openstack/neutron/src/branch/master/doc/source/install/compute-install-option2-rdo.rst09:07
mgoddardcan you maually enable ipv6 on the bridge via a sysctl?09:07
yoctozeptoso neutron recommends it09:08
yoctozeptomgoddard: yeah but it does not solve all cases09:08
yoctozeptoit only allows link-local then09:08
mgoddardhmm09:10
*** cz2 has joined #openstack-kolla09:10
mgoddardyoctozepto: what do you have for net.ipv6.conf.{default,all}.disable_ipv6?09:12
*** k_mouza has joined #openstack-kolla09:12
*** k_mouza has quit IRC09:12
mgoddardyoctozepto: https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/deployment/kernel/kernel-baremetal-puppet.yaml09:13
*** k_mouza has joined #openstack-kolla09:14
yoctozeptomgoddard: 0s09:14
mgoddardseems like libvirt has some broken logic somewhere09:15
yoctozeptoand this is not only about link-local so getting 0 on the bridge does not fix routable09:15
yoctozeptomgoddard: seems like kernel ;-)09:15
yoctozeptobridge should only bridge09:16
yoctozeptoiptables does it right09:16
yoctozeptoip6tables does not09:16
openstackgerritMichal Nasiadka proposed openstack/kolla master: Change fluentd to td-agent on CentOS  https://review.opendev.org/67561409:18
*** shyamb has quit IRC09:22
*** dougsz has quit IRC09:35
*** hjensas has joined #openstack-kolla09:37
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637609:50
*** dougsz has joined #openstack-kolla09:51
hrwmnasiadka: commented09:53
openstackgerritMerged openstack/kolla-ansible master: docs: ceph-ansible usage, add note around ceph keys  https://review.opendev.org/67793909:54
*** k_mouza has quit IRC09:54
*** zaitcev_ has quit IRC09:55
*** zaitcev_ has joined #openstack-kolla10:07
*** heikkine has joined #openstack-kolla10:07
mnasiadkahrw: thanks, all comments welcome :)10:07
*** shyamb has joined #openstack-kolla10:10
yoctozeptomgoddard: on controller+network node I have10:23
yoctozepto# sysctl net.bridge10:23
yoctozeptonet.bridge.bridge-nf-call-arptables = 010:23
yoctozeptonet.bridge.bridge-nf-call-ip6tables = 010:23
yoctozeptonet.bridge.bridge-nf-call-iptables = 010:23
yoctozeptoso something turned it off explicitly10:23
yoctozepto(or something did not turn it on explicitly?)10:24
yoctozeptojust installing docker seems to turn these on10:24
yoctozeptowell, installing and starting10:24
yoctozeptoso probably something did turn them off there10:24
yoctozeptothough that's no solution on compute nodes either way10:24
yoctozeptowonder what's the background anyway10:25
yoctozeptothis seems to conflict with aio approach10:25
yoctozeptoas you have a host which has to have them both on and off :D10:25
yoctozeptoSchroedinger's bridging10:25
yoctozeptohrw: have you any ubuntu/debian deployment and could you verify whether those are affected too? that would be interesting to know10:26
yoctozeptoannouncement: k-a ubuntu permanently broken10:34
yoctozeptoall jobs end in neutron bootstrap failure10:35
yoctozeptolet's call it a bad day :p10:35
hrwyoctozepto: can you rebase https://review.opendev.org/#/c/672700 one?10:35
yoctozeptohrw: sure, can do later10:36
yoctozeptonow onto meeting10:36
hrwyoctozepto: thanks10:36
yoctozepto(local one)10:36
*** shyamb has quit IRC10:41
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: Add CentOS/source aarch64 experimental job  https://review.opendev.org/66245610:44
hrwrebase only10:45
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: Add CentOS/source aarch64 experimental job  https://review.opendev.org/66245610:49
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: Add Debian/source aarch64 experimental job  https://review.opendev.org/55765910:49
*** k_mouza has joined #openstack-kolla10:54
*** shyamb has joined #openstack-kolla11:00
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637611:05
mgoddardyoctozepto: nova role should set those sysctls on computes11:10
mgoddardyoctozepto: yeah, noticing those ubuntu neutron failures. Ubuntu images were uploaded to dockerhub before centos, so perhaps we'll see failures everywhere soon?11:11
*** tonythomas has joined #openstack-kolla11:12
openstackgerritMichal Nasiadka proposed openstack/kolla master: Change fluentd to td-agent on CentOS  https://review.opendev.org/67561411:18
mnasiadkagod, this fluentd is killing me :D11:19
*** zaitcev_ has quit IRC11:20
openstackgerritMichal Nasiadka proposed openstack/kolla master: Change fluentd to td-agent on CentOS  https://review.opendev.org/67561411:22
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637611:25
*** kplant has joined #openstack-kolla11:29
*** zaitcev_ has joined #openstack-kolla11:32
hrwmnasiadka: +1 and let's see what zuul say11:35
openstackgerritMichal Nasiadka proposed openstack/kolla master: Change fluentd to td-agent on CentOS  https://review.opendev.org/67561411:36
*** shyamb has quit IRC11:41
*** shyam89 has joined #openstack-kolla11:41
*** k_mouza has quit IRC11:41
*** k_mouza has joined #openstack-kolla11:42
*** shyam89 has quit IRC11:45
*** k_mouza has quit IRC11:47
*** k_mouza has joined #openstack-kolla11:47
*** k_mouza has quit IRC11:52
*** dpawlik has quit IRC12:02
*** shyamb has joined #openstack-kolla12:03
openstackgerritMerged openstack/kolla-ansible stable/rocky: Moves monasca-thresh java.io.tmpdir to existing docker volume  https://review.opendev.org/67901312:04
shyambHi12:07
shyambIn ocata release, Is kolla ansible deployment containerized or not?12:08
hrwshyamb: kolla is container solution...12:09
shyambokay12:09
shyambWhat was the first openstack release that kolla-ansible supported?12:09
shyambhrw: ^^12:10
hrwshyamb: ocata was first when kolla-ansible was part of releases.openstack.org12:11
hrwI joined during pike/queens12:11
hrwshyamb: and I would not deploy <stein12:11
shyambhrw: great, thank you12:13
shyambOcata kolla not so stable?12:13
mgoddardshyamb: the project started around liberty12:14
shyambmgoddard: thanks for the info12:14
mgoddardshyamb: hrw is wrong to say do not deploy <stein - we have run pike, queens, rocky in production. If you are starting a new project though, stein or rocky is the best choice12:15
hrwright, meant new projects12:16
mgoddardwe currently maintain queens and later as stable branches12:16
mgoddardhrw: k12:16
yoctozepto<mgoddard> yoctozepto: yeah, noticing those ubuntu neutron failures. Ubuntu images were uploaded to dockerhub before centos, so perhaps we'll see failures everywhere soon?12:17
shyambmgoddard: Okay12:17
yoctozeptomgoddard: those seems to be from rebuilds12:17
yoctozeptoso centos should have the same source12:17
yoctozeptobut who knows :D12:17
shyambmgoddard: hrw: If someone wants to upgrade from ocata to queens, is it supported in kolla-ansible?12:18
mgoddardshyamb: yes, but no fast-forward. You need to upgrade to each major release12:18
mgoddardso ocata, pike, queens12:19
shyambmgoddard: Great12:24
shyambmgoddard: One of our client is using Ocata, so we need to support it12:25
shyambI am just checking how much it is possible12:25
shyambClient is running Ocata Kolla in production12:25
shyambI believe there are no major issues with it from installation and working perspective for our testing12:26
shyambif we need anything, we will always take first help from this channel12:26
openstackgerritMerged openstack/kolla-ansible stable/stein: Moves monasca-thresh java.io.tmpdir to existing docker volume  https://review.opendev.org/67901212:26
*** dpawlik has joined #openstack-kolla12:30
openstackgerritMichal Nasiadka proposed openstack/kolla master: Change fluentd to td-agent on CentOS  https://review.opendev.org/67561412:33
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637612:40
*** shyamb has quit IRC12:42
*** shyamb has joined #openstack-kolla12:47
*** shyamb has quit IRC12:52
*** watersj has joined #openstack-kolla12:52
openstackgerritMark Goddard proposed openstack/kolla-ansible master: WIP: Refactor service, endpoint and user registration  https://review.opendev.org/58759012:52
openstackgerritMichal Nasiadka proposed openstack/kolla master: Change fluentd to td-agent on CentOS  https://review.opendev.org/67561413:01
hrwmnasiadka: tell when it will be ready for review13:04
mnasiadkahrw: Zuul will say +1 then :D13:04
*** k_mouza has joined #openstack-kolla13:06
watersjyoctozepto, thanks for response yesterday. I do have a chicken and egg issue. My master node is down and won't start due to now container present. Yes I am to blame for pruning my container, is there way to get it started?13:10
*** dave-mccowan has joined #openstack-kolla13:10
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637613:12
*** k_mouza has quit IRC13:17
*** k_mouza has joined #openstack-kolla13:20
mnasiadkamgoddard: regarding https://review.opendev.org/#/c/673949/3 - maybe let's abandon my change and move communication to yours? :)13:25
*** keper7 has joined #openstack-kolla13:27
mgoddardmnasiadka: can do. I wasn't planning to complete it though :)13:30
mnasiadkamgoddard: well, there's no rush in this I guess - but it's a nice to have :)13:30
mgoddardmnasiadka: did you see my comment about the endpoint interface? I think it's a blocker for os_keystone_endpoint13:30
mnasiadkamgoddard: might be, the only idea I have to deliver interface to auth dictionary so os_client_config can use it13:31
mgoddardI don't think it takes it13:31
mgoddardI don't actually see where our module differentiates these two interfaces though13:33
*** itlinux has joined #openstack-kolla13:37
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637613:38
*** itlinux has quit IRC13:49
*** chason has joined #openstack-kolla13:50
*** priteau has joined #openstack-kolla13:55
openstackgerritOndrej Duchon proposed openstack/kolla-ansible master: Corrected notificaions for auditing.  https://review.opendev.org/67854414:00
*** k_mouza_ has joined #openstack-kolla14:02
*** k_mouza has quit IRC14:02
openstackgerritScott Solkhon proposed openstack/kolla-ansible master: Enable Swift Recon  https://review.opendev.org/67772014:07
*** k_mouza_ has quit IRC14:08
openstackgerritScott Solkhon proposed openstack/kolla-ansible master: Enable Swift Recon  https://review.opendev.org/67772014:12
*** openstackgerrit has quit IRC14:22
*** k_mouza has joined #openstack-kolla14:24
*** openstackgerrit has joined #openstack-kolla14:26
openstackgerritScott Solkhon proposed openstack/kolla master: Add effective and permitted capability to blackbox exporter  https://review.opendev.org/67715614:26
*** dpawlik has quit IRC14:29
*** hamdyk has quit IRC14:36
*** Luzi has quit IRC14:40
*** hjensas has quit IRC14:44
openstackgerritMerged x/kayobe stable/stein: Add support for Arista switches  https://review.opendev.org/67900914:46
*** skramaja has quit IRC14:50
mgoddardmeeting in 10 minutes14:51
*** hjensas has joined #openstack-kolla14:51
mgoddardmgoddard mnasiadka hrw egonzalez yoctozepto rafaelweingartne ^14:51
hrwsorry but can not attend today14:51
hrwhave to go and pick up some papers14:51
*** k_mouza has quit IRC14:53
mgoddardhrw: ok, np14:53
*** scottsol has joined #openstack-kolla14:55
*** igordc has joined #openstack-kolla14:57
*** scottsol has quit IRC14:58
*** scottsol has joined #openstack-kolla14:59
mgoddard#startmeeting kolla15:00
openstackMeeting started Wed Aug 28 15:00:43 2019 UTC and is due to finish in 60 minutes.  The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot.15:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:00
*** openstack changes topic to " (Meeting topic: kolla)"15:00
mgoddard#topic rollcall15:00
openstackThe meeting name has been set to 'kolla'15:00
*** openstack changes topic to "rollcall (Meeting topic: kolla)"15:00
mgoddard\o15:00
yoctozeptoo/15:01
scottsolo/15:01
priteau\o15:01
dougszLo15:02
*** dpawlik has joined #openstack-kolla15:02
yoctozeptodougsz: rectangular arms?15:03
chasono/15:03
dougszgood posture15:03
*** zaitcev_ has quit IRC15:04
mgoddardbod15:04
mgoddardbiceps?15:04
mgoddard#topic agenda15:04
yoctozeptohttps://www.urbandictionary.com/define.php?term=bod15:04
*** openstack changes topic to "agenda (Meeting topic: kolla)"15:04
yoctozeptodefinition 2?15:04
yoctozepto;o15:04
mgoddard* Roll-call15:05
mgoddard* Announcements15:05
mgoddard** PTL and TC nominations open for U cycle http://lists.openstack.org/pipermail/openstack-discuss/2019-August/008832.html15:05
mgoddard* Review action items from last meeting15:05
mgoddard* Kolla whiteboard https://etherpad.openstack.org/p/KollaWhiteBoard15:05
mgoddard* Kayobe Stein release status15:05
mgoddard* Train release planning15:05
mgoddard* Testing TLS15:05
mgoddard#topic announcements15:05
*** openstack changes topic to "announcements (Meeting topic: kolla)"15:05
mgoddard#info PTL and TC nominations open for U cycle15:05
mgoddard#link http://lists.openstack.org/pipermail/openstack-discuss/2019-August/008832.html15:05
mgoddardI will be running for kolla PTL again, but it is an election so please do feel free to run15:06
mgoddardAny other announcements?15:06
*** dpawlik has quit IRC15:07
mgoddard#topic Review action items from last meeting15:07
*** openstack changes topic to "Review action items from last meeting (Meeting topic: kolla)"15:07
mgoddardThere was one item last time15:07
mgoddardmgoddard to release stable branches15:07
mgoddardI did not15:07
mgoddard#action mgoddard to release stable branches15:08
mgoddard#topic Kolla whiteboard https://etherpad.openstack.org/p/KollaWhiteBoard15:08
*** openstack changes topic to "Kolla whiteboard https://etherpad.openstack.org/p/KollaWhiteBoard (Meeting topic: kolla)"15:08
mgoddardHow is CI looking? We had an issue with ubuntu earlier15:08
yoctozeptois it gone now?15:09
*** k_mouza has joined #openstack-kolla15:10
mgoddardzuul is being slwo15:11
mgoddardI'm not sure from looking at builds page, but there are quite a few failures recently15:11
mgoddardhard to tell if they are just broken patches though15:12
mgoddardI think we just need to keep an eye out for those jobs15:12
mgoddardall kolla-ansible-ubuntu-source jobs since  2019-08-28T07:39:52 have failed15:13
mgoddard#topic Kayobe Stein release status15:14
*** openstack changes topic to "Kayobe Stein release status (Meeting topic: kolla)"15:14
mgoddardI think there is just one more patch left that I'd like to land: https://review.opendev.org/67690715:15
openstackgerritMark Goddard proposed x/kayobe master: Set provisioning and cleaning networks as non-shared  https://review.opendev.org/67690715:15
*** zaitcev_ has joined #openstack-kolla15:16
mgoddard^ just fixed up a comment, hopefully it can now land15:16
mgoddardthen backport and we're god15:16
mgoddard*good15:16
mgoddardany other patches we're missing?15:16
mgoddard#topic Train release planning15:17
*** openstack changes topic to "Train release planning (Meeting topic: kolla)"15:17
mgoddard#info About 1 month until kolla feature freeze15:17
mgoddardso let's try to focus on those priorities that have been started15:18
mgoddardIn priority order:15:19
mgoddard* Nova cells v215:19
mgoddard* support matrix (actually this is above cells)15:19
mgoddard* support new ansible versions15:19
mnasiadkaFor support matrix is anybody volunteering to help me fill out the content for kolla?15:21
mgoddardgood question15:21
mgoddardcan it be automated?15:21
mgoddard* TLS everywhere15:22
mgoddardI guess we can easily generate the list of images15:23
mgoddardand maybe using the skip list we can generate lists of N/A?15:23
mgoddardothers might be harder15:23
yoctozeptomgoddard: good idea15:23
yoctozeptothen all not unbuildabe are at least C15:23
mgoddardyeah15:23
yoctozeptoand we promote others15:23
yoctozeptoas we see fit15:23
mgoddard+115:24
*** ivve has quit IRC15:24
mnasiadkaI’ll start populating from skip lists15:24
mnasiadkaAnd the rest is easy15:24
mnasiadkayoctozepto: working on k-a part?:)15:24
mgoddardthis seems to work15:25
mgoddardtools/build.py --list-images -t source -b ubuntu15:25
mgoddardat least I get different answers for ubuntu & centos15:26
mgoddardhopefully due to skip list15:26
yoctozeptomnasiadka: nope, sorry15:26
mgoddardanyway, I'll take an action to mark priority patches with RP+115:27
mgoddard#action mgoddard to mark Train priority patches ready for review with RP+115:27
mgoddardAnything else for Train?15:27
mgoddard#topic Testing TLS15:28
*** openstack changes topic to "Testing TLS (Meeting topic: kolla)"15:28
mgoddardIn https://review.opendev.org/#/c/664517 there is a new CI test being added with TLS enabled15:29
mgoddard#link https://review.opendev.org/#/c/664517/815:29
mgoddardping generalfuzz15:29
mgoddardI was thinking maybe it would be better to enable TLS in one of our existing tests15:30
mgoddarde.g. kolla-ansible-*-* or kolla-ansible-*-*-ceph15:30
mgoddardmaybe single node would make sense15:31
mgoddardso TLS for single node jobs, no TLS for multinode15:31
mgoddardwhat do people think?15:31
mgoddarddon't care?15:33
*** dpawlik has joined #openstack-kolla15:35
dougszjust wondering - why not TLS on multi-node?15:35
dougszIsn't that where it would be most useful?15:36
mgoddardcould do. Just thinking it's a longer job already, and TLS might make it even longer*15:36
mgoddard*untested15:36
dougszfair point15:37
mgoddardI could be convinced either way15:38
mgoddardok, let's move on15:38
mgoddard#topic open discussion15:38
*** openstack changes topic to "open discussion (Meeting topic: kolla)"15:38
*** dpawlik has quit IRC15:39
mgoddardDoes anyone have anything to discuss today?15:39
*** niceplace_ has joined #openstack-kolla15:39
dougszQuick one on the upgrade process:15:39
dougszWe have rolling upgrade, nova_safety_upgrade, legacy upgrade, and possibly fast forward upgrade15:40
dougszShould we simplify it and just support one?15:40
dougsz(this is for Nova)15:41
mgoddarddefault is rolling upgrade15:41
dougszyeah - that is the one that makes most sense to me, and should quite possibly have nova_safety_upgrade turned on as part of that15:41
mgoddardI'm not sure what legacy upgrade would be used for now15:42
mgoddardsafety upgrade is a tradeoff15:42
dougszas you say, it could be transformed into the fast upgrade method, but then, do we want so many upgrade paths which we don't test?15:42
mgoddardit gives you control plane downtime in exchange for no failed requests15:43
mgoddardI'm not sure about legacy upgrade. It was never quite clear to me why it was kept15:44
mgoddardfast forward upgrades aren't supported yet, but I do wonder whether we'd use the legacy upgrade code for that15:45
openstackgerritOndrej Duchon proposed openstack/kolla-ansible master: Corrected notificaions for auditing.  https://review.opendev.org/67854415:45
mgoddardhowever, given we don't support FFU now and have no one working on it it's not much of an argument15:45
mgoddardI'd like to keep the safety upgrade default, it's more in the spirit of a rolling upgrade15:46
dougszyeah, was wondering if we actually *want* to support FFU. I like the idea of one upgrade path for Nova, which is the one we test.15:46
mgoddardFFU is useful if you're stuck on an old release15:47
dougszas in safety upgrade off?15:47
mgoddardyes15:47
mgoddardit's not really unsafe without it, you just might get some API request failures if they use new features15:48
mgoddardbut if you turn your API off then you can't service any requests, which is kind of like a failure15:48
mgoddardI guess I'm up for removing the legacy upgrade path in kolla-ansible15:49
dougszok, thanks, that would be one path gone at least15:49
mgoddardany other topics?15:50
generalfuzzsorry - just joined15:51
mgoddardhi generalfuzz15:51
mgoddardwe 'discussed' the TLS job but didn't really get any comments15:51
generalfuzzregarding TLS - We can re-org the test suite whatever way makes most sense15:51
mgoddardsure. I think just getting something to pass would be a good first step15:52
generalfuzzyes, and we are a bit off from that still15:52
mgoddardmy comment about using upstream ansible modules, I don't think we'll get to that this cycle so I'd suggest fixing the ones we have. They're bundled in the kolla_toolbox image in the kolla repo15:53
mgoddardanything else on TLS?15:55
generalfuzz I am running into issues enabling TLS internally, and  believe that the "validate_certs: "{{ kolla_verify_tls | bool}}" isn't fully propegated to openstack python libraries15:55
mgoddardyes, I think that's the issue15:55
mgoddardyou could compare the modules in kolla_toolbox with those in upstream ansible. I think they now use openstacksdk directly rather than shade15:56
mgoddardok, let's leave it there15:57
mgoddard#endmeeting15:57
*** openstack changes topic to "Topic for #openstack-kolla is: support: ask.openstack.org | New to Kolla: docs @ https://docs.openstack.org/kolla/latest/ | | Kolla IRC meetngs on Wednesdays @ 15:00 UTC - see agenda @ https://goo.gl/OXB0DL - IRC channel is *LOGGED* @ http://goo.gl/3mzZ7b"15:57
openstackMeeting ended Wed Aug 28 15:57:28 2019 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)15:57
openstackMinutes:        http://eavesdrop.openstack.org/meetings/kolla/2019/kolla.2019-08-28-15.00.html15:57
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/kolla/2019/kolla.2019-08-28-15.00.txt15:57
openstackLog:            http://eavesdrop.openstack.org/meetings/kolla/2019/kolla.2019-08-28-15.00.log.html15:57
yoctozeptomgoddard: re: https://bugzilla.redhat.com/show_bug.cgi?id=174612315:57
openstackbugzilla.redhat.com bug 1746123 in libvirt "Isolated IPv6 bridge network does not work with br_netfilter being on" [Unspecified,New] - Assigned to libvirt-maint15:57
yoctozeptoDaniel helped me locate the problem for the bare case15:57
yoctozeptoturns out firewalld was the culprit15:58
yoctozeptoforgot to check the raw tables15:58
yoctozeptoipv6 rp_filter applied in there matched exactly the behavior I saw15:58
yoctozeptonow onto why kolla's data plane ipv6 is broken even without firewalld...15:59
*** zaitcev_ has quit IRC16:01
*** k_mouza has quit IRC16:03
*** scottsol has quit IRC16:04
yoctozeptowatersj: best apply this patch onto your kolla-ansible16:05
yoctozeptothen run  mariadb_recovery16:05
*** zaitcev_ has joined #openstack-kolla16:13
watersjyoctozepto, I already took sledgehammer approach. I have small test cluster and redeploying now should be up within 30minutes. After my ha tests again, I won't prune the comntainers.16:16
*** itlinux has joined #openstack-kolla16:17
*** itlinux is now known as itlinux-away16:20
*** watersj has quit IRC16:25
*** itlinux-away has quit IRC16:25
*** iclon has quit IRC16:32
*** keper7 has quit IRC16:38
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637616:39
*** jonaspaulo has joined #openstack-kolla16:39
*** keper7 has joined #openstack-kolla16:40
mnasiadkayoctozepto: got the neutron error once again :)16:41
mnasiadkaI guess it's officially a CI blocker in kolla-ansible...16:41
*** k_mouza has joined #openstack-kolla16:45
*** k_mouza has quit IRC16:49
openstackgerritDoug Szumski proposed openstack/kolla-ansible master: Minimal support for deploying multiple instances of MariaDB  https://review.opendev.org/61975616:52
openstackgerritDoug Szumski proposed openstack/kolla-ansible master: Copy Nova role as a basis for the Nova cell role  https://review.opendev.org/67565816:52
openstackgerritDoug Szumski proposed openstack/kolla-ansible master: [WIP] Factor out nova-cell role  https://review.opendev.org/67565916:52
*** dougsz has quit IRC17:01
*** itlinux has joined #openstack-kolla17:02
*** itlinux has quit IRC17:07
*** zaitcev_ has quit IRC17:12
*** k_mouza has joined #openstack-kolla17:12
*** k_mouza has quit IRC17:17
*** zaitcev_ has joined #openstack-kolla17:24
*** Chaserjim has joined #openstack-kolla17:46
*** zaitcev__ has joined #openstack-kolla17:55
*** chason has quit IRC17:56
*** zaitcev_ has quit IRC17:58
*** priteau has quit IRC18:12
*** igordc has quit IRC18:23
*** k_mouza has joined #openstack-kolla18:40
*** itlinux has joined #openstack-kolla18:44
*** k_mouza has quit IRC18:45
*** dciabrin has quit IRC18:47
*** itlinux has quit IRC18:48
*** zaitcev_ has joined #openstack-kolla18:51
yoctozepto<mnasiadka> I guess it's officially a CI blocker in kolla-ansible...18:54
yoctozepto:<<<18:54
*** zaitcev__ has quit IRC18:55
mnasiadkayoctozepto: but it’s showing up sometimes :)18:55
mnasiadkaDon’t be sad, drink a beer ;)18:55
yoctozeptomnasiadka: I'm a water drinker ;-)19:05
yoctozepto(I know, too bad)19:05
mnasiadkayoctozepto: naah, drink a lot of water :)19:17
*** zaitcev_ has quit IRC19:18
*** ivve has joined #openstack-kolla19:20
*** itlinux has joined #openstack-kolla19:22
openstackgerritMerged x/kayobe master: Set provisioning and cleaning networks as non-shared  https://review.opendev.org/67690719:24
keper7Hey all, this blueprint talks about setting up bridge ports to allow an aio deploy with a single nic. https://blueprints.launchpad.net/kolla-ansible/+spec/kolla-ansible-aio-single-interface-prep  - Can anyone point me in the right direction as to how I can manually make them? I'm trying to use Kolla-Ansible to deploy Openstack on an Intel Nuc with just one port19:26
*** itlinux_ has joined #openstack-kolla19:28
*** zaitcev_ has joined #openstack-kolla19:30
*** itlinux has quit IRC19:31
openstackgerritMark Goddard proposed x/kayobe stable/stein: Set provisioning and cleaning networks as non-shared  https://review.opendev.org/67912919:34
*** dpawlik has joined #openstack-kolla19:36
*** dpawlik has quit IRC19:41
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: Modernize a way of configuring Docker daemon  https://review.opendev.org/67270019:43
*** kplant has quit IRC19:46
*** itlinux has joined #openstack-kolla20:06
*** dciabrin has joined #openstack-kolla20:06
*** atmark has quit IRC20:07
*** mloza has joined #openstack-kolla20:08
*** itlinux_ has quit IRC20:09
*** zaitcev_ has quit IRC20:17
*** itlinux has quit IRC20:18
*** itlinux has joined #openstack-kolla20:20
*** zaitcev_ has joined #openstack-kolla20:29
*** itlinux has quit IRC20:33
*** itlinux has joined #openstack-kolla20:33
*** itlinux has quit IRC21:00
*** itlinux has joined #openstack-kolla21:04
*** k_mouza has joined #openstack-kolla21:11
*** itlinux has quit IRC21:12
*** itlinux has joined #openstack-kolla21:12
*** zaitcev_ has quit IRC21:16
*** k_mouza has quit IRC21:16
*** itlinux has quit IRC21:22
*** k_mouza has joined #openstack-kolla21:34
*** dpawlik has joined #openstack-kolla21:37
*** k_mouza has quit IRC21:38
*** dpawlik has quit IRC21:41
*** jbadiapa has quit IRC21:52
*** ivve has quit IRC22:03
*** jonaspaulo has quit IRC22:24
*** k_mouza has joined #openstack-kolla23:03
*** k_mouza has quit IRC23:08
*** k_mouza has joined #openstack-kolla23:26
*** k_mouza has quit IRC23:30
*** keper7 has quit IRC23:36
*** keper7 has joined #openstack-kolla23:37
*** k_mouza has joined #openstack-kolla23:53
*** k_mouza has quit IRC23:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!