Wednesday, 2019-07-10

*** altlogbot_3 has joined #openstack-kolla00:28
*** cgrosjean has quit IRC00:32
*** zbr has quit IRC00:34
*** altlogbot_3 has quit IRC00:37
*** zbr has joined #openstack-kolla00:44
*** zhanglong has joined #openstack-kolla00:57
*** altlogbot_1 has joined #openstack-kolla01:06
*** altlogbot_1 has quit IRC01:15
*** altlogbot_3 has joined #openstack-kolla02:18
*** irclogbot_3 has joined #openstack-kolla02:18
*** altlogbot_3 has quit IRC02:25
*** irclogbot_3 has quit IRC02:26
*** factor has quit IRC02:34
*** factor has joined #openstack-kolla02:35
*** k_mouza has joined #openstack-kolla03:10
*** Sravan has joined #openstack-kolla03:14
*** k_mouza has quit IRC03:15
*** Sravan has quit IRC03:38
*** irclogbot_3 has joined #openstack-kolla04:07
*** Sravan has joined #openstack-kolla04:10
*** shyamb has joined #openstack-kolla04:11
*** irclogbot_3 has quit IRC04:12
*** altlogbot_0 has joined #openstack-kolla04:13
*** jschlueter has quit IRC04:13
*** Sravan has quit IRC04:15
*** altlogbot_0 has quit IRC04:17
*** shyamb has quit IRC04:20
*** shyamb has joined #openstack-kolla04:20
openstackgerritzhulingjie proposed openstack/kolla-cli master: Sync Sphinx requirement  https://review.opendev.org/57763604:23
openstackgerritzhulingjie proposed openstack/kolla-cli master: Sync Sphinx requirement  https://review.opendev.org/57763604:29
*** factor has quit IRC04:36
*** shyamb has quit IRC04:37
*** whoami-rajat has joined #openstack-kolla04:52
*** altlogbot_2 has joined #openstack-kolla05:02
*** dannins has quit IRC05:03
*** irclogbot_2 has joined #openstack-kolla05:07
*** jschlueter has joined #openstack-kolla05:07
*** mjturek has quit IRC05:20
*** mjturek has joined #openstack-kolla05:21
*** altlogbot_2 has quit IRC05:21
*** irclogbot_2 has quit IRC05:22
*** skramaja has joined #openstack-kolla05:30
*** janki has joined #openstack-kolla05:33
*** pcaruana has joined #openstack-kolla05:37
openstackgerritMerged openstack/kolla-cli master: Sync Sphinx requirement  https://review.opendev.org/57763605:39
*** Luzi has joined #openstack-kolla05:43
openstackgerritzhulingjie proposed openstack/kolla-cli master: Remove the doc/build  https://review.opendev.org/66998105:44
*** shyamb has joined #openstack-kolla05:45
yoctozeptomorning channel05:48
*** mjturek has quit IRC05:50
*** mjturek has joined #openstack-kolla05:51
*** lei-zh has joined #openstack-kolla05:56
*** lei-zh has quit IRC05:58
*** lei-zh has joined #openstack-kolla05:59
*** shyamb has quit IRC06:02
*** shyamb has joined #openstack-kolla06:02
*** hjensas has quit IRC06:03
openstackgerritMerged openstack/kolla-cli master: Remove the doc/build  https://review.opendev.org/66998106:06
*** shyamb has quit IRC06:06
*** cah_link has joined #openstack-kolla06:07
*** shyamb has joined #openstack-kolla06:14
*** pcaruana has quit IRC06:20
*** lei-zh1 has joined #openstack-kolla06:29
*** lei-zh has quit IRC06:31
*** shyamb has quit IRC06:35
*** luksky11 has joined #openstack-kolla06:37
*** janki has quit IRC06:40
*** cah_link has quit IRC06:44
*** dpawlik has joined #openstack-kolla06:44
*** janki has joined #openstack-kolla06:57
*** ivve has joined #openstack-kolla07:02
*** altlogbot_0 has joined #openstack-kolla07:03
*** rpittau|afk is now known as rpittau07:04
stingrayzamorning yoctozepto & everyone :)07:04
*** lei-zh1 has quit IRC07:05
*** lei-zh1 has joined #openstack-kolla07:06
*** altlogbot_0 has quit IRC07:08
*** lei-zh has joined #openstack-kolla07:08
yoctozeptomorning stingrayza07:09
*** shyamb has joined #openstack-kolla07:10
*** lei-zh1 has quit IRC07:10
*** serhatd has joined #openstack-kolla07:12
*** hamdyk has joined #openstack-kolla07:30
openstackgerritgugug proposed openstack/kolla-cli master: Move the stestr command into testenv section  https://review.opendev.org/66999907:33
openstackgerritgugug proposed openstack/kolla-cli master: Move the stestr command into testenv section  https://review.opendev.org/66999907:37
*** irclogbot_0 has joined #openstack-kolla07:39
openstackgerritgugug proposed openstack/kolla-cli master: Move the stestr command into testenv section  https://review.opendev.org/66999907:39
openstackgerritwangwei proposed openstack/kolla master: Add partition detection to fix osd initialization failure  https://review.opendev.org/67000107:42
*** irclogbot_0 has quit IRC07:43
openstackgerritgugug proposed openstack/kolla-cli master: Add doc8 test into pep8  https://review.opendev.org/67000307:45
openstackgerritgugug proposed openstack/kolla-cli master: Add doc8 test into pep8  https://review.opendev.org/67000307:47
*** Wasaac has joined #openstack-kolla07:52
*** pcaruana has joined #openstack-kolla07:54
*** Wasaac has quit IRC07:55
*** Wasaac has joined #openstack-kolla07:56
*** shyamb has quit IRC07:57
*** Wasaac has quit IRC07:58
*** Wasaac has joined #openstack-kolla07:58
*** hjensas has joined #openstack-kolla08:01
*** shyamb has joined #openstack-kolla08:05
openstackgerritgugug proposed openstack/kolla-cli master: Add doc8 test into pep8  https://review.opendev.org/67000308:08
openstackgerritChason Chan proposed openstack/kolla-ansible master: Fix the incorrect backup_driver configuration  https://review.opendev.org/67000608:11
*** gkadam has joined #openstack-kolla08:13
*** umbSublime has quit IRC08:13
*** umbSublime has joined #openstack-kolla08:16
*** priteau has joined #openstack-kolla08:17
*** k_mouza has joined #openstack-kolla08:18
ohwhyosaMorning!08:22
*** gkadam has quit IRC08:22
openstackgerritgugug proposed openstack/kolla-cli master: Move the stestr command into testenv section  https://review.opendev.org/66999908:24
openstackgerritgugug proposed openstack/kolla-cli master: Move the stestr command into testenv section  https://review.opendev.org/66999908:25
*** cgrosjean has joined #openstack-kolla08:38
ohwhyosaOkay, I think I have it pretty close08:38
ohwhyosaIt all deploys good but the network, networking works not08:39
ohwhyosaOpenVSwitch is not creating all the bridges08:39
ohwhyosaAnd then sometimes after some reboots/ docker restarts, it does bring them up08:40
ohwhyosaBut still cant usually ping from qrouter to the world and cirros doesn't get proper ips08:40
ohwhyosaOnce it did get an internal one08:41
yoctozeptoohwhyosa: anything in the logs?08:42
*** janki has quit IRC08:43
ohwhyosayoctozepto: Yup, mainly saying that the interface/bridge doesn't exists08:45
ohwhyosaDo you prefer one pastebin with logs/globals/inventory or separate pastebins?08:45
ohwhyosaThe architecture, by tthe way, is one deploy, one compute-networking, one storage and one networking; all of them with two Gb ethernet cables to a switch without vlans08:46
ohwhyosaThe only "extra" thing is kibana, magnum and skydive08:47
yoctozeptoohwhyosa: separate reads better usually08:48
ohwhyosaPerfect!08:50
ohwhyosaindeed, yoctozepto do you prefer pastebin or is there other service you like to share this kind of stuff?08:50
*** janki has joined #openstack-kolla08:51
*** dougsz has joined #openstack-kolla08:51
ohwhyosayoctozepto: inventory: https://pastebin.com/hK8ihp9Y08:53
ohwhyosayoctozepto: globals.yml https://pastebin.com/nEYEyVmp08:54
*** shyamb has quit IRC08:54
*** irclogbot_1 has joined #openstack-kolla08:57
ohwhyosayoctozepto: neutron-openvswitch agent log https://pastebin.com/kTQpyVhB08:57
openstackgerritKrzysztof Klimonda proposed openstack/kolla-ansible master: Internal OpenStack endpoints encryption spec  https://review.opendev.org/66386508:58
openstackgerritMerged openstack/kolla-cli master: Add doc8 test into pep8  https://review.opendev.org/67000308:58
mnasiadkamorning08:58
ohwhyosayoctozepto: neutron-l3-agent https://pastebin.com/WzfveLkR08:58
ohwhyosamorning!08:59
ohwhyosayoctozepto: neutron metadata (though this one seems not too important, this rabbit mq errors usually go away after a restart) https://pastebin.com/HJBnFSQN09:00
*** irclogbot_1 has quit IRC09:02
ohwhyosayoctozepto: ovsdb_server log https://pastebin.com/Q33Tq5ND (the address already in use kills me a bit inside, sometimes it appears after a destroy-deploy, sometimes it doesn't09:04
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: ceph-nfs: Add rpcbind to Ubuntu host bootstrap  https://review.opendev.org/66931509:05
ohwhyosayoctozepto: neutron-server log https://pastebin.com/ZVPpCAL409:06
*** janki has quit IRC09:07
ohwhyosayoctozepto: And I think that's it!  sorry if it's too much, I added them to be thorough and put the title in the comment so you can search it09:07
ohwhyosayoctozepto: Need anything like ip netns show, ovs-vsctl show or something else? And thanks a lot, again!09:08
*** janki has joined #openstack-kolla09:09
yoctozeptoohwhyosa: name='shit' ?09:11
*** altlogbot_1 has joined #openstack-kolla09:11
yoctozeptohttps://pastebin.com/kTQpyVhB09:11
yoctozeptolmao09:11
openstackgerritwangwei proposed openstack/kolla master: Add partition detection to fix osd initialization failure  https://review.opendev.org/67000109:12
*** shyamb has joined #openstack-kolla09:13
ohwhyosa<yoctozepto "https://pastebin.com/kTQpyVhB"> I am a bit beyond my reasoning skills with the networks09:13
*** altlogbot_1 has quit IRC09:14
ohwhyosaIn my attempt number 900000 I ended up calling it shit instead of demo1 and normal stufff like that hahahah sorry09:14
*** mjturek has quit IRC09:15
*** mjturek has joined #openstack-kolla09:16
yoctozepto[network]09:16
yoctozepto192.168.200.11  ansible_become=true  ansible_python_interpreter=/usr/bin/python309:16
yoctozepto09:16
yoctozepto09:16
yoctozepto[compute]09:16
yoctozepto192.168.200.11  ansible_become=true  ansible_python_interpreter=/usr/bin/python209:16
yoctozeptothis is interesting09:16
yoctozeptopython2/3 on the same host ;p09:16
yoctozeptowonder what ansible did to that host var09:17
ohwhyosaAHHH! I forgot to change it afterwards!! Might that be it? Changing it right now, will report further (though I believe it failed before i did that, and i believe logs all mention python2)09:18
ohwhyosaThanks!09:18
yoctozeptothe most worrying is the 2019-07-10T09:00:17.979Z|03972|ovsdb_jsonrpc_server|ERR|ptcp:6640:127.0.0.1: listen failed: Address already in use09:19
*** k_mouza has quit IRC09:19
yoctozeptodestroy may have been unclean09:19
stingrayzaohwhyosa, yoctozepto: a colleague of mine had a similar issue - he accidentally installed openvswitch on the _host_ machine - and it automatically started getting in the way of the ovs docker containers09:23
mgoddardmorning09:23
*** lei-zh has quit IRC09:25
ohwhyosastingrayza: I may be having that problem as well, thanks!09:25
*** altlogbot_3 has joined #openstack-kolla09:25
ohwhyosaOkay, will fix the python thingie and the openvswitch one!09:25
ohwhyosaBy the way, rocky release doesn't work with python3, right?09:25
yoctozeptomorning mgoddard09:25
ohwhyosao/09:26
*** Wasaac has quit IRC09:26
openstackgerritcaoyuan proposed openstack/kolla-cli master: update the tox minversion to 2.0  https://review.opendev.org/62622509:27
*** Wasaac has joined #openstack-kolla09:27
stingrayzaohwhyosa: good luck. the rule of thumb I'm trying to work with is to give a _clean/minimal_ install over to kolla-ansible to do everything. anything kolla-ansible needs, it should be installing for itself.09:27
*** altlogbot_3 has quit IRC09:28
ohwhyosastingrayza: Yup, I actually did reinstall the servers a couple times (we are going bare metal instead of vms now) with min config09:28
*** altlogbot_3 has joined #openstack-kolla09:29
*** Wasaac has quit IRC09:29
*** mjturek has quit IRC09:29
*** Wasaac has joined #openstack-kolla09:29
*** priteau has quit IRC09:30
openstackgerrityanghongyu proposed openstack/kolla-cli master: Blacklist sphinx 2.1.0 (autodoc bug)  https://review.opendev.org/67002209:30
*** lei-zh has joined #openstack-kolla09:31
*** Wasaac has quit IRC09:31
*** k_mouza has joined #openstack-kolla09:31
*** Wasaac has joined #openstack-kolla09:32
*** Wasaac has quit IRC09:34
*** altlogbot_3 has quit IRC09:34
*** Wasaac has joined #openstack-kolla09:34
yoctozeptomgoddard: kolla-cli is experiencing its renaissance09:35
*** Wasaac has quit IRC09:36
*** Wasaac has joined #openstack-kolla09:37
*** irclogbot_1 has joined #openstack-kolla09:37
*** Wasaac has quit IRC09:38
*** Wasaac has joined #openstack-kolla09:39
*** irclogbot_1 has quit IRC09:40
*** Wasaac has quit IRC09:41
*** Wasaac has joined #openstack-kolla09:42
openstackgerrityanghongyu proposed openstack/kolla-cli master: Blacklist sphinx 2.1.0 (autodoc bug)  https://review.opendev.org/67002209:42
*** Wasaac has quit IRC09:43
*** Wasaac has joined #openstack-kolla09:44
*** altlogbot_3 has joined #openstack-kolla09:45
*** Wasaac has quit IRC09:46
*** Wasaac has joined #openstack-kolla09:46
*** serhatd has quit IRC09:47
*** serhatd has joined #openstack-kolla09:47
*** Wasaac has quit IRC09:48
*** Wasaac has joined #openstack-kolla09:49
*** altlogbot_3 has quit IRC09:50
*** Wasaac has quit IRC09:51
*** Wasaac has joined #openstack-kolla09:51
egonzalezohwhyosa are you using something outside default in the network config?09:52
*** Wasaac has quit IRC09:53
egonzalezas yoctozepto said,error is likely ovs is already running on the host, can be a system service or a stale container09:54
*** Wasaac has joined #openstack-kolla09:54
egonzalezalso, maybe another unkown process bind to that port09:54
ohwhyosa<egonzalez "ohwhyosa are you using something"> Nope, tried with opendaylight and dpdk, but they failed miserably for me and decided to go a bit more baby-steps paced09:55
ohwhyosa<egonzalez "as yoctozepto said,error is like"> Now it seems it deployed correctly09:55
ohwhyosaLemme check some networking stuff09:56
*** Wasaac has quit IRC09:56
yoctozeptoegonzalez: actually that was said by stingrayza but I mentioned the unclean destroy09:56
*** Wasaac has joined #openstack-kolla09:56
ohwhyosaHmmm, it does create the bridges, it does create the netns, but the qrouter netns still cant ping google09:57
egonzalezcan the qrouter ping the gateway?09:58
yoctozeptoohwhyosa: maybe it should not be able to ;p09:58
egonzalezis the port created as router gateway or subnet interface?09:58
*** Wasaac has quit IRC09:58
yoctozeptothat and whether the gateway can reach google09:58
ohwhyosaI'm using the initrunconce for quick testing09:58
openstackgerritRaimund Hook (stingrayza) proposed openstack/kolla-ansible master: Update designate-guide cli command for dns_domain  https://review.opendev.org/67002409:59
egonzalezneed to adapt the script to your public address config, first lines09:59
*** Wasaac has joined #openstack-kolla09:59
* ohwhyosa sent a long message: < https://matrix.org/_matrix/media/v1/download/matrix.org/hzhqNZiKuAvPpXoKoUulFUIb >09:59
ohwhyosahmmmm09:59
ohwhyosaThat might be it09:59
egonzalezso your public network info is correct to conect outside10:00
*** VickyG has quit IRC10:00
ohwhyosaIn this case of a local-only deployment, my public address would just be one in the same range  as internal_vip?10:00
*** Wasaac has quit IRC10:00
yoctozeptoohwhyosa: this is attached to the external bridge10:00
*** Wasaac has joined #openstack-kolla10:01
ohwhyosato the external bridge i attached eno2, but it said not to assign it ny ip, I believe?10:01
egonzalezohwhyosa in example, in my dev env, neutron_external_interface is attached to a NATed kvm network10:01
egonzalezinterface no IP, but neutron network configured in the kvm network range10:02
*** Wasaac has quit IRC10:03
ohwhyosaso, if the network that can access the gateway (in this case a real router) is 192.168.200.0/2410:03
ohwhyosait should be in that range, same as internal_vip_address but a different address??10:03
*** Wasaac has joined #openstack-kolla10:03
egonzalezyes, configure neutron gateway to point the router, and ensure to give a pool of free addresses or will easily consume all the range10:04
ohwhyosaEXT_NET_CIDR='192.168.200.0/24'10:04
ohwhyosaEXT_NET_RANGE='start=192.168.200.110,end=192.168.200.300'10:04
ohwhyosaEXT_NET_GATEWAY='192.168.200.1'10:04
ohwhyosalike so? ^10:04
egonzalezexcept 192.168.200.300, hard to get an IP 300 in a 244 range ;)10:05
*** Wasaac has quit IRC10:05
*** Wasaac has joined #openstack-kolla10:06
ohwhyosaDuh10:06
ohwhyosaToo little sleep hahahaha10:06
ohwhyosayoctozepto: egonzalez Again thanks a lot!10:06
egonzalezbtw, if need help with ODL let me know10:06
ohwhyosaI'll check and report back in a sec10:06
ohwhyosaGreat! If this goes well, next step will be opendaylighting10:07
ohwhyosaSo thanks for the offer10:07
ohwhyosainitrunonce does not work with ODL, right?10:07
*** Wasaac has quit IRC10:08
ohwhyosaBut I can use the commands there still, good for practicing10:08
*** Wasaac has joined #openstack-kolla10:08
egonzalezit works10:09
egonzalezodl is just the neutron backend, any neutron action should work10:10
*** Wasaac has quit IRC10:10
*** Wasaac has joined #openstack-kolla10:11
*** Wasaac has quit IRC10:12
*** Wasaac has joined #openstack-kolla10:13
openstackgerritMerged openstack/kolla-cli master: Blacklist sphinx 2.1.0 (autodoc bug)  https://review.opendev.org/67002210:14
ohwhyosaWell, when using odl as a neutron driver (with default options) it threw some errors (don't have them at hand, but next deployment I'll be sure to keep them)10:14
ohwhyosaquick question regarding var names in initrunonce10:14
ohwhyosain allocation-pool you use EXT_NET_RANGE and in range you use EXT_NET_CIDR or should it be the other way around?10:14
ohwhyosaI mean, the script does it that way, but it confuses me a bit10:15
*** Wasaac has quit IRC10:15
*** Wasaac has joined #openstack-kolla10:15
*** Wasaac has quit IRC10:17
*** Wasaac has joined #openstack-kolla10:18
ohwhyosaHmmm it now looks like properly configured in that regard, but i can't ping the gateway from the router ns10:18
ohwhyosadestination host unreachable10:19
*** heikkine has quit IRC10:19
*** Wasaac has quit IRC10:20
*** Wasaac has joined #openstack-kolla10:20
*** zhanglong has quit IRC10:22
*** shyamb has quit IRC10:22
ohwhyosaFrom the machine (eth 0 with 10.0.0.16) i can't ping the gateway (according to the routes, 10.0.0.2)10:22
ohwhyosaalso can directly ping the 169.254.169.25410:22
*** Wasaac has quit IRC10:22
*** Wasaac has joined #openstack-kolla10:23
*** Wasaac has quit IRC10:23
*** irclogbot_0 has joined #openstack-kolla10:23
*** Wasaac has joined #openstack-kolla10:23
ohwhyosabut i can't ping extetrnal10:23
ohwhyosafrom the router namespace i can ping 10.0.0.210:24
ohwhyosathe router itself is on 10.0.0.1 and on 192.168.200.11710:24
ohwhyosathe host can ping 192.168.200.1, but not the router namespace10:25
*** Wasaac has quit IRC10:25
*** Wasaac has joined #openstack-kolla10:26
*** irclogbot_0 has quit IRC10:26
ohwhyosaOkay, i localized it with eno2 being down10:26
ohwhyosaNow it can ping the gateway10:26
ohwhyosait can thus also ping google10:26
ohwhyosaand the machien10:26
ohwhyosaGUYS I LOVE YOU10:27
*** Wasaac has quit IRC10:27
ohwhyosaI've been like 2 weeks trying to make this work (granted my general network lack of understanding and openstack in general)10:27
ohwhyosanow it works AND i learnt a lot10:28
ohwhyosanext step, ODL, and next to that trying to use vlans or vxlans10:28
*** Wasaac has joined #openstack-kolla10:28
*** Wasaac has quit IRC10:30
*** lei-zh has quit IRC10:30
*** lei-zh has joined #openstack-kolla10:30
*** Wasaac has joined #openstack-kolla10:30
*** Wasaac has quit IRC10:32
*** Wasaac has joined #openstack-kolla10:33
*** Wasaac has quit IRC10:35
*** Wasaac has joined #openstack-kolla10:35
*** lei-zh has quit IRC10:35
*** Wasaac has quit IRC10:37
*** Wasaac has joined #openstack-kolla10:38
*** Wasaac has quit IRC10:39
*** hamdyk has quit IRC10:40
*** Wasaac has joined #openstack-kolla10:40
*** Wasaac has quit IRC10:42
stingrayzaohwhyosa: The first six months I spent with Openstack were one of the steepest learning curves I've ever encountered. But the only way to learn the ropes is to roll your sleeves up and start trying to _make it work_. You're in the middle of that - so you're in the right place ;)10:42
*** Wasaac has joined #openstack-kolla10:42
*** Wasaac has quit IRC10:44
*** hamdyk has joined #openstack-kolla10:44
stingrayza(and this channel/group seems to be particularly helpful/welcoming - I remember posting in another openstack channel in irc a couple of years back as a complete newbie and not getting much help)10:45
*** Wasaac has joined #openstack-kolla10:45
*** shyamb has joined #openstack-kolla10:45
*** Wasaac has quit IRC10:47
ohwhyosastingrayza Yeah, this channel really does seem like a light at the end of the tunnel10:47
*** Wasaac has joined #openstack-kolla10:47
ohwhyosaAnd I understand people being tired of hearing the same questions once and again, with people like me lost beyond hope hahaha So it does make it all the more amazing the amount of help I've been given here10:48
ohwhyosaYou people go well beyond the call of duty10:48
*** Wasaac has quit IRC10:49
*** Wasaac has joined #openstack-kolla10:50
*** Wasaac has quit IRC10:52
*** Wasaac has joined #openstack-kolla10:52
*** janki has quit IRC10:55
*** shyam89 has joined #openstack-kolla11:03
*** shyamb has quit IRC11:04
mnasiadkayoctozepto: so now we need a kolla-cli dashboard I guess? :D11:04
egonzalezis mariadb still failing rocky-stein upgrade? just tested and everything worked fine for me11:09
egonzalez*almost no data, haproxy on top, 3 centos source containers11:10
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Add 'allow *' to getting ceph mds keyring  https://review.opendev.org/67003511:11
*** Wasaac has quit IRC11:15
*** Wasaac has joined #openstack-kolla11:16
*** kplant has joined #openstack-kolla11:18
*** Wasaac has quit IRC11:18
*** Wasaac has joined #openstack-kolla11:19
*** dougsz has quit IRC11:19
*** Wasaac has quit IRC11:20
*** Wasaac has joined #openstack-kolla11:21
*** Wasaac has quit IRC11:23
*** Wasaac has joined #openstack-kolla11:23
mgoddardegonzalez: it usually works, but we are still seeing occasional failures11:24
*** Wasaac has quit IRC11:25
*** Wasaac has joined #openstack-kolla11:26
*** Wasaac has quit IRC11:28
*** Wasaac has joined #openstack-kolla11:28
kplantmgoddard: totally missed your comment 1832878 - my bad11:29
*** Wasaac has quit IRC11:30
*** Wasaac has joined #openstack-kolla11:31
mnasiadkamgoddard: occasional failures is what I see in ceph mds as well :)11:32
*** Wasaac has quit IRC11:32
*** Wasaac has joined #openstack-kolla11:33
*** Wasaac has quit IRC11:35
*** Wasaac has joined #openstack-kolla11:35
*** Wasaac has quit IRC11:37
*** Wasaac has joined #openstack-kolla11:38
*** Wasaac has quit IRC11:40
*** Wasaac has joined #openstack-kolla11:40
*** Wasaac has quit IRC11:42
*** Wasaac has joined #openstack-kolla11:43
ohwhyosawhat is the process with reconfigure? Lemme see if I got that right: 1.) on deploy host, create /etc/kolla/config/magnum/magnum.conf11:43
ohwhyosa2.) add [cinder] and the content i want to add in that section11:43
ohwhyosa3.) kolla-ansible -i multinode reconfigure -t magnum11:44
ohwhyosa4.) Restart the containers11:44
ohwhyosaI did that and it is not changing stuff, so i probably got it wrong11:44
*** Wasaac has quit IRC11:45
ohwhyosaAlso tried /etc/kolla/config/magnum-api and magnum-conductor11:45
*** Sravan has joined #openstack-kolla11:45
*** Wasaac has joined #openstack-kolla11:45
*** Wasaac has quit IRC11:47
*** Wasaac has joined #openstack-kolla11:48
*** Wasaac has quit IRC11:49
*** Sravan has quit IRC11:49
*** Wasaac has joined #openstack-kolla11:50
*** Wasaac has quit IRC11:52
ohwhyosaOh, looking at the playbook, it seems it has to either be on: /etc/kolla/config/global.conf, /etc/kolla/config/magnum.conf, /etc/kolla/config/magnum/magnum-api.conf or /etc/kolla/config/magnum/inventory_hostname/magnum.conf, is that right?11:52
*** Wasaac has joined #openstack-kolla11:52
*** Wasaac has quit IRC11:54
kplantso for the cinder section of magnum conf, you want: /etc/kolla/config/magnum.conf11:55
*** Wasaac has joined #openstack-kolla11:55
kplant[kplant@chrnc-dev-os-build-01 cloud-instance-build]$ cat /etc/kolla/config/magnum.conf11:55
kplant[cinder]11:55
kplantdefault_docker_volume_type = standard11:55
*** Wasaac has quit IRC11:57
*** Wasaac has joined #openstack-kolla11:58
*** heikkine has joined #openstack-kolla11:59
ohwhyosakplant: Great! I did in the end inside magnum/magnum-api.conf and magnum-conductor.conf, after reading the playbook!11:59
ohwhyosaworked amazingly, thanks!11:59
ohwhyosaFor some reason i thought it required an intermediate folder like /etc/kolla/config/magnum/magnum.conf11:59
*** Wasaac has quit IRC11:59
*** Wasaac has joined #openstack-kolla12:00
*** Wasaac has quit IRC12:02
*** Wasaac has joined #openstack-kolla12:02
*** Wasaac has quit IRC12:04
egonzalezohwhyosa btw, reconfigure will also restart the containers, not need to do it aftewards12:04
*** shyamb has joined #openstack-kolla12:05
*** Wasaac has joined #openstack-kolla12:05
kplant^12:06
*** Wasaac has quit IRC12:07
kplantand i believe it only restarts the containers impacted by the configuration change12:07
kplantwith or without tags supplied12:07
*** Wasaac has joined #openstack-kolla12:07
*** shyam89 has quit IRC12:08
*** Wasaac has quit IRC12:09
*** Wasaac has joined #openstack-kolla12:10
*** Wasaac has quit IRC12:11
*** Wasaac has joined #openstack-kolla12:12
*** Wasaac has quit IRC12:14
*** Wasaac has joined #openstack-kolla12:14
*** Wasaac has quit IRC12:16
*** Wasaac has joined #openstack-kolla12:17
*** Wasaac has quit IRC12:19
*** Wasaac has joined #openstack-kolla12:19
*** Wasaac has quit IRC12:21
*** Wasaac has joined #openstack-kolla12:22
*** Wasaac has quit IRC12:23
*** Wasaac has joined #openstack-kolla12:24
yoctozepto<ohwhyosa> For some reason i thought it required an intermediate folder like /etc/kolla/config/magnum/magnum.conf12:25
yoctozeptohmm, it should12:25
yoctozeptoto conform with the others12:25
yoctozeptolemme check12:25
yoctozepto<mnasiadka> yoctozepto: so now we need a kolla-cli dashboard I guess? :D12:26
kplanti had initially tried that but didn't work for me until i moved magnum.conf into /etc/kolla/config/12:26
yoctozeptoseems so12:26
*** Wasaac has quit IRC12:26
*** Wasaac has joined #openstack-kolla12:26
yoctozeptoah, ok, it's magnum global config12:27
yoctozeptolike for the others but you usually configure the specific service12:27
stingrayzamgoddard: I've added Kayobe to lgtm.com (https://lgtm.com/projects/g/openstack/kayobe). The good news is that there's no _major_ issues, although there is a couple of warnings. I'll take a look and see if there's anything I feel confident to submit a patch against.12:27
yoctozeptoso it would be /etc/kolla/config/nova.conf for global12:27
yoctozeptoyet /etc/kolla/config/nova/nova-api.conf for api12:27
yoctozeptoanalogously magnum12:27
kplantls -lah12:28
kplanterrr12:28
kplantwindow focus is important12:28
*** Wasaac has quit IRC12:28
yoctozeptokplant: nah, let's run bash in irc12:28
yoctozeptocowsay "You look great today"12:29
*** Wasaac has joined #openstack-kolla12:29
yoctozeptooops ;D12:29
kplant-bash: cowsay: command not found12:29
stingrayzayoctozepto: :D12:29
kplanti have a funny short story about cowsay...12:30
kplantfor our bastions at work i had written a motd that would change daily and it was basically `fortune | cowsay`12:30
kplantbut it was supposed to be sfw.... yeah not so much, some VPs were copied on some e-mails12:30
kplanti wonderif i can find the fortune that was generated for that day12:31
*** Wasaac has quit IRC12:31
*** Wasaac has joined #openstack-kolla12:31
*** mchlumsky has joined #openstack-kolla12:33
*** Wasaac has quit IRC12:33
*** Wasaac has joined #openstack-kolla12:34
kplanthttps://pastebin.com/cG10XCX9 <-- they didn't appreciate that one12:35
*** Wasaac has quit IRC12:35
*** cah_link has joined #openstack-kolla12:35
*** Wasaac has joined #openstack-kolla12:36
*** Wasaac has quit IRC12:38
mgoddardstingrayza: I wasn't aware of lgtm.com, looks interesting12:38
*** Wasaac has joined #openstack-kolla12:38
stingrayzathey have most of the openstack projects already mapped. I know one of the guys there, got him to add kayobe...12:39
*** Wasaac has quit IRC12:40
*** Wasaac has joined #openstack-kolla12:41
mgoddardlooks like a few easy wins, and some more subtle things12:42
yoctozeptostingrayza: hahaha12:43
yoctozeptosorry, meant kplant12:43
yoctozeptobut you can laugh too anyway12:44
stingrayza:)12:44
yoctozeptostingrayza: you have experience with lgtm?12:44
kplant:D12:45
kplantthe security analysis seems very appealing12:46
stingrayzayoctozepto: only a little - only really because a buddy works there. less so defining queries. they've recently posted a new blog on using their query language against python: https://blog.semmle.com/python-code-analysis-ql/12:46
*** mchlumsky has quit IRC12:47
yoctozeptostingrayza: skimmed over it, but it really looks appealing12:47
*** priteau has joined #openstack-kolla12:47
stingrayzakplant, yoctozepto: I remember the first time one of our QA guys ran our ansible playbooks. he came back to the dev team with a "this ansible thing you have is very unprofessional" and showed us his console logs - and his ansible had very kindly picked up the fact he had cowsay installed and used it in all the output :)12:47
stingrayzayoctozepto: yeah - I think it needs a clean mind and a bit of focus to get into it.12:48
kplanthahahaha12:48
yoctozeptostingrayza: yeah, I saw that12:48
yoctozeptotruly blurs the output12:48
kplantstingrayza: someone had a sense of humor when writing their ansible.cfg12:48
ohwhyosastingrayza yoctozepto kplant They don't like it when I put cowsay either, they say it "polutes the ansible stdout"12:48
yoctozeptofun as it is, I think they might really be hurting their looks ;p12:48
ohwhyosaMore like it makes it awesome, but hey12:49
yoctozepto<kplant> stingrayza: someone had a sense of humor when writing their ansible.cfg12:49
yoctozeptoit picks it up12:49
yoctozeptoBY DEFAULT12:49
yoctozeptoiirc12:49
kplantwhoa whoa WHAT12:49
kplantreally!?12:49
stingrayzayes it does - by DEFAULT!12:49
ohwhyosaEspecially with random cows12:49
yoctozeptoyes12:49
kplantthat's amazing12:49
yoctozeptoas soon as you have it installed12:49
kplanti'm putting cowsay on my build node immediately12:49
yoctozeptoit's not installed by default though12:49
yoctozeptoso usually you are safe12:49
yoctozeptololz12:49
*** mchlumsky has joined #openstack-kolla12:49
kplanti think we should put cowsay in requirements.txt12:50
kplant+1 ?12:50
yoctozeptokplant: first try it out12:50
yoctozeptoyou'll change your mind ;p12:50
kplanti definitely wasn't being serious12:51
yoctozeptothough iirc ansible logs are still free of cowsay12:51
*** henriqueof has joined #openstack-kolla12:51
kplantansible output is hard enough to read without filters12:51
kplant500 lines of output without linefeeds is strenuous enough12:51
yoctozepto<kplant> ansible output is hard enough to read without filters12:51
yoctozeptothe truth has been spoken12:51
priteauhttps://github.com/ansible/ansible/issues/1053012:52
yoctozepto-vvv adds more line breaks12:52
kplantceph-ansible is kind enough to include some goodies to parse '\\n'12:52
yoctozeptothough it adds more blur too12:52
yoctozeptopriteau,12:52
yoctozeptoLabels12:52
yoctozeptofeature12:52
priteauLooks like the Ansible core developers are happy with the way things are with regards to cowsay12:52
yoctozeptopriteau: maybe it's their plan to fire unsuspecting devops guys12:54
kplantaah, the silent purge12:54
*** dougsz has joined #openstack-kolla12:57
yoctozeptothe only issue with lgtm is that it's post-merge for us13:00
yoctozeptobecause it obviously does not integrate with gerrit13:00
yoctozepto;D13:00
ohwhyosausually the stdout_callback = yaml helps with reading13:02
ohwhyosaIt returns a yaml complete with linebreaks13:03
ohwhyosathough there was a case in which it stopped working on an installation of mine, for some reason13:03
yoctozeptoohwhyosa: you are a savior, did not know this one, looks promising13:04
*** skramaja has quit IRC13:04
kplantohwhyosa: yeah that's what ceph-ansible does13:04
yoctozeptoand it's 2.5+13:04
kplantit's very easy on the eyes compared to default13:04
yoctozeptoguys, we need this in CI ;D13:05
*** irclogbot_1 has joined #openstack-kolla13:06
*** irclogbot_1 has quit IRC13:08
*** altlogbot_1 has joined #openstack-kolla13:11
*** altlogbot_1 has quit IRC13:16
*** shyamb has quit IRC13:17
*** irclogbot_3 has joined #openstack-kolla13:17
*** irclogbot_3 has quit IRC13:22
*** shyamb has joined #openstack-kolla13:27
stingrayzayoctozepto: I was just about to suggest the same :) `stdout_callback = debug` or `stdout_callback = yaml` both work nicely13:28
*** goldyfruit has joined #openstack-kolla13:28
henriqueofSorry if I missed something but is there any news on the stein release?13:30
stingrayzayoctozepto: see more here: https://github.com/ansible/ansible/issues/2707813:30
goldyfruito/13:30
ohwhyosaHi!13:31
mgoddardhenriqueof: no release yet. We have hit a few more mariadb issues, I don't really have time to debug right now13:31
ohwhyosayoctozepto: It is one of those quality of life little ansible tricks13:31
ohwhyosaSo you can add again your beautiful cowsay13:32
henriqueofmgoddard: where is the problem described? Mybe I can take a look.13:32
ohwhyosaanyone here uses magnum?13:32
goldyfruito/13:32
mgoddardhenriqueof: we see occasional issues with mariadb during upgrade13:33
mgoddardsome notes here: https://etherpad.openstack.org/p/kolla-ci-errors13:33
goldyfruitohwhyosa, shoot :)13:33
mgoddardhttp://zuul.openstack.org/builds?job_name=kolla-ansible-centos-source-upgrade-ceph&branch=stable%2Fstein13:34
henriqueofi see.13:35
henriqueofThanks!13:35
mgoddardTBH we seem to see various similar issues on rocky in CI too, perhaps it's not such a blocker13:35
henriqueofmgoddard: The the problems accour when upgrading right? Anyproblems related to new deployments?13:36
yoctozepto<mgoddard> henriqueof: we see occasional issues with mariadb during upgrade13:36
*** altlogbot_2 has joined #openstack-kolla13:36
yoctozeptoduring deploy too13:36
mgoddardgit example, this one failed during rocky deploy: http://logs.openstack.org/08/669808/2/check/kolla-ansible-centos-source-upgrade-ceph/ac264fa/primary/logs/ansible/deploy13:36
*** altlogbot_2 has quit IRC13:36
henriqueofyoctozepto: ah, okay.13:36
ohwhyosagoldyfruit: I'm trying to access the dashboard that it creates, following the magnum docs (i did the eval and stuff)13:36
ohwhyosaI can see there is a kube-dashboard service13:36
ohwhyosaBut for some reason I'm not able to access it13:37
goldyfruitSo you ran the kubectl-proxy command right ?13:37
ohwhyosayup13:37
* ohwhyosa sent a long message: < https://matrix.org/_matrix/media/v1/download/matrix.org/JgXrQymMZRkNKTZVoDMPjvCy >13:38
goldyfruitWhich OpenStack release ?13:38
ohwhyosaPlease tell me that sent as a single message13:38
ohwhyosarocky!13:38
goldyfruitohwhyosa, just got one :)13:38
goldyfruitThe kubernetes-dashboard pod is running not restarting ?13:38
ohwhyosaCluster with 1 master and 3 minion13:38
ohwhyosayup, running, gimme a sec13:39
* ohwhyosa sent a long message: < https://matrix.org/_matrix/media/v1/download/matrix.org/aswINUYmiFHXNLoHTSQpLnvY >13:39
goldyfruitOk, so running13:40
goldyfruitwhat about your security group ?13:40
ohwhyosaand if i do {proxy:port}/ui it just sends me to the api13:40
goldyfruitI did a commit that fixed the communication between master and minion13:40
goldyfruithttps://storyboard.openstack.org/#!/story/200529413:41
*** Wasaac has quit IRC13:41
ohwhyosaIt is freshly deployed, as in, I did the magnum cluster template, created the cluster, it failed because of network invalid auth, did it again, it worked and there it is13:42
ohwhyosaMeaning, i did nothing at all apart from deploying and trying to access the api (and getting a bit familiarized with the kubectl)13:42
*** Wasaac has joined #openstack-kolla13:42
mgoddardhenriqueof: how is your galera-fu?13:42
goldyfruitjust to be sure, open the security groups13:42
goldyfruitohwhyosa, wide open!13:42
henriqueofmgoddard: pretty bad, my skills are mostly in programming and linux in general.13:43
ohwhyosagoldyfruit: oh, you mean the openstack security groups?13:44
goldyfruitohwhyosa, yep13:44
ohwhyosahmmm so i need to allow port 844313:45
ohwhyosaand then kubectl proxy --port=9090 and http://localhost:9090/ui13:46
ohwhyosa?13:46
goldyfruitohwhyosa, the issue I got was the master not able to reach the minion13:47
goldyfruitegonzalez, could you please guide me about howto fix this: https://review.opendev.org/#/c/616050/13:53
goldyfruitI don't get "masakari ansible roles is not in depends-on"13:53
*** whoami-rajat has quit IRC13:54
*** whoami-rajat has joined #openstack-kolla13:54
yoctozeptogoldyfruit: commit message - add a tag near Change-Id - Depends-On: https://review.opendev.org/ID_OF_MASAKARI_CHANGE13:56
*** BjoernT has joined #openstack-kolla13:56
goldyfruityoctozepto, ohhhhhhhhhhh13:56
goldyfruitWas looking into kolla-ansible xD13:56
*** Wasaac has quit IRC13:57
yoctozeptothis is so that Zuul picks up the needed stuff13:58
*** Wasaac has joined #openstack-kolla13:58
goldyfruitGood to know13:58
openstackgerritGaëtan Trellu proposed openstack/kolla-ansible master: Testing Masakari role in gate  https://review.opendev.org/61605013:58
*** Wasaac has quit IRC13:59
*** Wasaac has joined #openstack-kolla14:00
*** Wasaac has quit IRC14:02
*** Wasaac has joined #openstack-kolla14:02
*** cah_link has quit IRC14:03
*** Wasaac has quit IRC14:04
*** Wasaac has joined #openstack-kolla14:05
*** hamzaachi has joined #openstack-kolla14:09
*** shyam89 has joined #openstack-kolla14:12
goldyfruitohwhyosa, is it ok ?14:15
*** shyamb has quit IRC14:16
*** hamzaachi has quit IRC14:16
ohwhyosagoldyfruit: I'm trying14:16
ohwhyosaIt is supposedly up14:16
ohwhyosaI did open more ports on the security group, though apparently it is exposing everything through 644314:17
ohwhyosaand kubectl cluster-info says14:17
ohwhyosaKubernetes master is running at https://192.168.200.111:644314:17
ohwhyosaCoreDNS is running at https://192.168.200.111:6443/api/v1/proxy/namespaces/kube-system/services/kube-dns14:17
ohwhyosakubernetes-dashboard is running at https://192.168.200.111:6443/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard14:17
ohwhyosathough going to that url is unauthorized14:17
goldyfruitThe issue is not from your laptop and the master14:17
ohwhyosaand kubectl proxy doesn't have any ui14:18
ohwhyosaas in14:18
goldyfruitit's between the master and the minion14:18
ohwhyosayeah, I opened in the security groups both ingress and egress 844314:18
goldyfruitwhich is why I recommend you just for the test to ofully pen TCP/UDP in both security groups14:18
goldyfruitJust to eliminate this possibility14:18
ohwhyosaGood, thanks!14:19
ohwhyosaIs it 8443 or 6443?14:19
ohwhyosaI did open 8443 on both minion and master14:19
goldyfruit844314:20
goldyfruit6443 is the Kube API14:20
ohwhyosagood, good14:20
ohwhyosaall is open then14:20
openstackgerritMerged openstack/kolla-ansible master: Add 'allow *' to getting ceph mds keyring  https://review.opendev.org/67003514:21
*** Wasaac has quit IRC14:26
*** Wasaac_ has joined #openstack-kolla14:26
ohwhyosagoldyfruit: doing kubectl describe po {kube_dashboard_pod}14:28
ohwhyosait says that kubedashboard port is 9090/tcp14:29
ohwhyosaI should open that one right?14:29
goldyfruitkubectl describe deployment kubernetes-dashboard  -n kube-system14:29
goldyfruitNow if you run kubectl -n kube-system get service kubernetes-dashboard14:32
*** hamzaachi has joined #openstack-kolla14:32
goldyfruityou should have a cluster-ip and a port14:32
ohwhyosa-n doesn't let me do anything, I have to use --namespace (go figure)14:32
goldyfruityou could try to connect on your master node14:32
ohwhyosaBut in general14:32
ohwhyosaI did a get deployments (because that one did not appear)14:32
ohwhyosaand this is the only deployment it sees14:32
goldyfruitand run curl -k https://CLUSTER-IP14:32
ohwhyosacoredns   1         1         1            1           2h14:33
goldyfruitI remember having issue with basic Rocky deployment and Magnum14:33
goldyfruitIf I remember well it's k8s 1.9.8 with Flannel14:33
goldyfruitAnd we got issue with Flannel14:33
ohwhyosayup, I'm using flannel (it had calico, too, but i went with the defaults)14:35
ohwhyosaCLUSTER-IP is Kubernetes master is running at https://192.168.200.111:6443 this one right?14:35
goldyfruitYou git 6443 with kubectl describe deployment kubernetes-dashboard  -n kube-system ?14:36
goldyfruityou got*14:36
ohwhyosano, nope14:37
ohwhyosathe deployment it doesn't allow me14:37
ohwhyosaBecause tere is no such deployment14:37
ohwhyosabut describing services I get14:37
ohwhyosa10.254.177.12314:37
*** dpawlik has quit IRC14:37
*** Luzi has quit IRC14:38
goldyfruitohwhyosa, we are using the Magnum version from Stein into our Rocky deployment14:38
goldyfruitto get the latest updated14:38
ohwhyosaI should do that from inside the kubernetes cluster right? else I have no route to host14:39
goldyfruitthat way we are able to run k8s 1.14 and calico14:39
goldyfruitI guess this is why I got so much differences with your setup14:39
*** altlogbot_1 has joined #openstack-kolla14:39
goldyfruitYes you should do that into the kube-master14:39
ohwhyosagoldyfruit: makes sense! (by the way, should the flannel ip be on the same network as neutron external or that is a completely virtualized docker network?)14:40
ohwhyosaWill tell you in a sec the result of the curl14:40
goldyfruitnop the cluster ip is internal to kubernetes14:41
goldyfruityoctozepto, I tried to install masakari client into centos7 via pip and got this error: https://paste.api-zulu.com/inajozofel.php14:42
*** Wasaac_ has quit IRC14:42
*** altlogbot_1 has quit IRC14:42
*** Wasaac has joined #openstack-kolla14:42
goldyfruitmissing gcc14:42
goldyfruitinstalling gcc and python-devel and the installation passed14:44
*** shyam89 has quit IRC14:44
*** hamdyk has quit IRC14:46
ohwhyosaDo the keys generated by initrunonce have a default password?14:49
ohwhyosaNope, it seems to be coreos14:50
mgoddardMeeting in 10 minutes14:50
mgoddardmgoddard mnasiadka hrw egonzalez yoctozepto ^14:50
*** altlogbot_1 has joined #openstack-kolla14:51
ohwhyosagoldyfruit: Mind if I write you dm as to not pollute the meeting?14:51
goldyfruitsure14:52
*** altlogbot_1 has quit IRC14:56
mgoddardok let's go15:00
mgoddard#startmeeting kolla15:00
openstackMeeting started Wed Jul 10 15:00:39 2019 UTC and is due to finish in 60 minutes.  The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot.15:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:00
*** openstack changes topic to " (Meeting topic: kolla)"15:00
openstackThe meeting name has been set to 'kolla'15:00
mgoddard#topic rollcall15:00
*** openstack changes topic to "rollcall (Meeting topic: kolla)"15:00
mgoddard\o15:00
chasono/15:00
Wasaaco/15:01
kplanto'15:01
*** gtarnaras has joined #openstack-kolla15:03
mgoddardI think this meeting conflicts with yoctozepto's commute or dinner arrangements - he's always active either side of it but not during :)15:03
mgoddardmnasiadka: around?15:03
mgoddard#topic agenda15:04
*** openstack changes topic to "agenda (Meeting topic: kolla)"15:04
mgoddard* Roll-call15:04
mgoddard* Announcements15:04
mgoddard** Welcome yoctozepto, latest core reviewer15:04
mgoddard** Removed zhubingbing and Duong Ha-Quang from core reviewer list15:04
mgoddard** Kayobe being added as a kolla deliverable https://review.opendev.org/66929915:04
mgoddard** From next week, meeting will cover kayobe15:04
mgoddard* Review action items from last meeting15:04
mgoddard* Kolla whiteboard https://etherpad.openstack.org/p/KollaWhiteBoard15:04
mgoddard* Stein release status15:04
mgoddard* Train release planning15:04
mgoddard#topic announcements15:04
*** openstack changes topic to "announcements (Meeting topic: kolla)"15:04
mgoddard#info Welcome yoctozepto, latest core reviewer15:04
mgoddard#info Removed zhubingbing and Duong Ha-Quang from core reviewer list15:04
mgoddard#info Kayobe being added as a kolla deliverable15:04
mgoddard#link https://review.opendev.org/66929915:04
mnasiadkao/ (sorry for being late)15:05
mgoddard#info From next week, meeting will cover kayobe15:05
mgoddardhi mnasiadka, np15:05
mgoddardAny other announcements?15:05
mgoddardI should probably make one more15:05
yoctozeptoo/15:05
yoctozepto(sorry too)15:05
mgoddard#info PTL (mgoddard) expecting a baby, due date 28th July. Will be away for ~3 weeks paternity leave15:06
mgoddardhi yoctozepto15:06
mgoddardI expect to be logging in from time to time to check in15:07
mgoddard#topic Review action items from last meeting15:07
*** openstack changes topic to "Review action items from last meeting (Meeting topic: kolla)"15:07
chasonmgoddard, congrats!15:07
*** altlogbot_1 has joined #openstack-kolla15:07
mgoddard:) thanks chason15:07
mgoddardNo action items last time15:07
yoctozeptomgoddard: congrats! bless you all15:07
mgoddard#topic Kolla whiteboard https://etherpad.openstack.org/p/KollaWhiteBoard15:07
*** openstack changes topic to "Kolla whiteboard https://etherpad.openstack.org/p/KollaWhiteBoard (Meeting topic: kolla)"15:08
mgoddard:)15:08
yoctozepto<goldyfruit> yoctozepto, I tried to install masakari client into centos7 via pip and got this error: https://paste.api-zulu.com/inajozofel.php15:08
yoctozeptonope, I was on time the last time15:08
mgoddardLet's check recent CI15:08
yoctozeptosorry, wronggg pasteee15:08
yoctozepto<mgoddard> I think this meeting conflicts with yoctozepto's commute or dinner arrangements - he's always active either side of it but not during :)15:08
mgoddard#link https://etherpad.openstack.org/p/KollaWhiteBoard15:08
yoctozepto<yoctozepto> nope, I was on time the last time15:08
yoctozepto(as for goldyfruit I believe the CI issue is somewhere else)15:09
mgoddardyoctozepto: that's true, my apologies15:10
mgoddardCI mostly green15:10
yoctozeptoyeah, we did a good job15:10
mgoddard#topic Stein release status15:10
*** openstack changes topic to "Stein release status (Meeting topic: kolla)"15:10
mgoddardSo close15:10
mgoddardI even pushed a review to create a new RC15:11
mgoddardBut yoctozepto found some more mariadb failures, during deploy and upgrade15:11
*** altlogbot_1 has quit IRC15:12
yoctozeptolet's wait this week, I'll spend some time digging into mariadb issues - let's release anyways if we cannot fix them now, quality is already much much better15:12
mgoddardFWIW, we also see these issues on other branches15:12
mgoddard(well at least rocky)15:12
yoctozeptomgoddard: yup, as if we are doing something wrong - though we are doing one thing wrong: docker stop15:12
yoctozeptothis class of errors should be gone15:12
yoctozeptoafter really waiting for mysql shutdown15:12
yoctozepto(partially amended by 60s timer)15:13
yoctozeptothough the other problem classes remain15:13
yoctozeptomgoddard suggested it could be due to missing haproxy15:13
mgoddardI saw the 'WSREP not ready' issue in rocky earlier15:13
yoctozepto(is anyone else sans me and Mark taking part in this meeting? ;p )15:13
mgoddardyoctozepto: generally it is me and one other person doing the talking, if I'm lucky :D15:14
yoctozepto;D15:14
mgoddardI was trying to think if there is a way we could get haproxy involved without using an overlay15:15
mgoddardcould disable keepalived, and configure a 'VIP' on the primary15:15
yoctozeptofwiw, the deadlocks seem to be unrelated to mariadb per se, most likely the way we run things or the way things run themmselves15:15
mgoddard(manually)15:15
mgoddardor just use api_interface_address15:15
mgoddardand separate frontend and backend port15:15
kplantmgoddard: how is that handled when the primary goes down?15:16
kplantif vrrp isn't used to move the vip15:16
mgoddardkplant: doesn't really matter in CI15:16
yoctozeptokplant: we don't take down primary for cluster testing15:16
*** mjturek has joined #openstack-kolla15:16
kplantoh ok. i'll sit back down :]15:16
yoctozeptokplant: no no, please share15:17
mgoddardremain standing kplant :)15:17
kplanti just meant for the current topic, no worries15:17
mgoddarddoes that plan sound crazy? it does depend on being able to configure separate a listening port for all services we test, not sure if that's possible15:18
mgoddardoh, it's not supported in rocky. That was the blocker, remember now15:18
mgoddardso wouldn't work for upgrade jobs15:18
mgoddardmaybe we do need an overlay then15:18
yoctozeptohas there been any testing around overlay?15:19
yoctozeptobelieve egonzalez was trying out something?15:19
mgoddardthat doesn't sound like something we could do quickly though15:19
*** irclogbot_1 has joined #openstack-kolla15:19
yoctozeptonope, definitely15:19
yoctozeptoand there is no guarantee it fixes mariadb15:19
mgoddardegonzalez was looking at multinode haproxy, I don't think he tried overlay yet15:19
mgoddardnope15:19
mgoddardok, yoctozepto said he would spend some time looking at mariadb this week, I will try to help where I can.15:21
yoctozeptoyeah, but no promises, I'm no mariadb-guru15:21
mgoddarddefining N jobs to run in parallel is quite good for testing flakey things15:22
yoctozeptowish I were15:22
yoctozeptoas in ceph case15:22
*** irclogbot_1 has quit IRC15:22
yoctozeptohigher chance of failing15:22
mgoddardtempting to just add that sleep back - perhaps we're just not waiting for the right thing15:22
yoctozeptomgoddard: 60 sleep will help with one class of problems for sure15:23
mgoddardright15:23
yoctozeptobut there are some which did not have the mariadb killed15:23
mgoddardI remember priteau saying that order of shutdown matters, perhaps that is something to investigate15:23
yoctozeptoyet wsrep does not want to work later15:23
yoctozeptoas in; it works, works and does not15:23
priteaumgoddard: well, my understanding is that order of shutdown helps to reliably keep the same node as the most advanced.15:24
priteauBut I haven't experimented with enough shutdowns to draw a conclusion from it.15:25
mgoddardpriteau: I find it hard to see how you could reliably determine the most advanced, especially if they are all the same at the point you check15:25
priteauIf you shut down all but one replica, wouldn't the remaining one be the most advanced?15:26
mgoddardwhat if one you shut down was more advanced when you shut it down15:27
mgoddarddon't know enough about galera to know what it would do15:28
*** henriqueof has quit IRC15:29
yoctozeptomgoddard: so far galera has proven to work reverse to what I would expect it too15:29
yoctozeptoHF instead of HA15:29
mgoddardhow do you mean?15:29
mgoddard:)15:29
yoctozeptoI left a note in CI errors15:29
yoctozeptocopy-pasting:15:29
yoctozepto^ these should be fixable by waiting for mariadb to really shutdown - else a slave might not want to recover because it is behind (seems broken to me but that's how life goes: https://stackoverflow.com/questions/54664565/unable-to-complete-sst-transfer-due-to-wsrep-sst-position-cant-be-set-in-past )15:29
yoctozeptomight be my English being bad15:30
yoctozeptoor my time understanding15:30
yoctozeptobut it does not make sense to me receive such error15:30
yoctozeptois not that what galera should do15:31
yoctozepto"I'm older, got newer, let's sync"15:31
*** altlogbot_3 has joined #openstack-kolla15:31
yoctozeptoinstead it is: "fsck, they are newer, bailing out lol"15:31
mgoddardcould be that they diverged?15:32
yoctozeptobut this one we probably fix by not forcibly crashing the containers15:32
mgoddardi.e. both have newer than common root15:32
mgoddardthat should help15:32
yoctozeptono idea, honestly15:32
yoctozeptoand nothing to trigger that either15:32
yoctozeptowe get that whenever we forcibly kill the process15:33
*** quantomworks has joined #openstack-kolla15:33
yoctozeptonot otherwise15:33
yoctozeptoand moreover it's from a slave afaik15:33
*** BjoernT_ has joined #openstack-kolla15:33
mgoddardI did think at one point we should check docker logs for the message where it has a stop timeout and fail the job15:33
yoctozeptomaybe we have some bug in mariadb config causing this oddity15:34
yoctozepto(thinking loudly)15:34
*** altlogbot_3 has quit IRC15:34
yoctozeptomgoddard: but there was none15:34
*** ivve has quit IRC15:34
yoctozeptoit kills them silently as far as CI logs go15:35
yoctozeptoor I missed something15:35
mgoddarddocker journal I think has something15:35
mgoddardalternatively we stop using 'docker stop' and replace with 'docker kill' and a manual poll, then fail if it doesn't stop15:36
yoctozeptoJul 08 21:53:20 primary dockerd[10281]: time="2019-07-08T21:53:20.701738294Z" level=debug msg="Sending kill signal 15 to container 90a7c430eac1ee06f1f804346c43dd0c6f9d3da714383eb57547ba23e6c443f4"15:36
yoctozeptodon't remember what 15 stands for15:36
*** BjoernT has quit IRC15:36
mgoddardsigterm15:36
mgoddardkill is 915:36
*** BjoernT has joined #openstack-kolla15:36
openstackgerritGaëtan Trellu proposed openstack/kolla master: Add HAcluster containers  https://review.opendev.org/66876515:36
yoctozeptothen odd15:36
mgoddardno message 10 seconds later?15:37
yoctozeptogot it15:37
yoctozeptoJul 08 22:14:31 primary dockerd[10281]: time="2019-07-08T22:14:31.250643659Z" level=debug msg="Sending kill signal 9 to container d8b893da786b16fa32a5ecfef30f12dcb5f2713765986d8c93a01ba420781ec2"15:37
yoctozeptothe log did not download fully yet, sorry15:37
yoctozeptohttp://logs.openstack.org/30/669730/2/check/kolla-ansible-centos-source-upgrade-ceph/6f71867/primary/logs/system_logs/docker.txt.gz15:38
*** BjoernT_ has quit IRC15:38
yoctozepto^ 4 of 915:38
yoctozeptoi.e. 4 times kill -915:38
mgoddardgrep 'failed to exit within' /var/log/docker.log15:38
mgoddardor simialr15:38
yoctozepto3 matches somehow15:39
*** hjensas has quit IRC15:39
yoctozeptoone was a real DELETE15:39
*** irclogbot_3 has joined #openstack-kolla15:39
yoctozeptoJul 08 22:15:50 primary dockerd[10281]: time="2019-07-08T22:15:50.381314870Z" level=debug msg="Calling DELETE /v1.39/containers/nova_consoleauth?force=True&link=False&v=False"15:39
yoctozeptoJul 08 22:15:50 primary dockerd[10281]: time="2019-07-08T22:15:50.381409920Z" level=debug msg="Sending kill signal 9 to container 5b78a8437161a5cadff4a9421495e88c51ff388268bad27110ede84e2bef827f"15:39
mgoddardlet's move on. we can continue with this after the meeting15:40
yoctozeptoyeah, exactly15:40
mgoddard#topic Train release planning15:40
*** openstack changes topic to "Train release planning (Meeting topic: kolla)"15:40
mgoddardWe seem so wrapped up in stein, I'm worried we're ignoring Train, so thought I'd add this topic15:40
mgoddardWe have lots of features on the whiteboard https://etherpad.openstack.org/p/KollaWhiteBoard15:41
mgoddardSome big ones without owners15:41
mgoddardCentOS8 / CentOS py3 depends on CentOS 8, out of our hands for now15:41
openstackgerritGaëtan Trellu proposed openstack/kolla-ansible master: WIP: Add HAcluster Ansible role  https://review.opendev.org/67010415:41
mgoddardalthough some work was done using fedora to test py3 tripleo15:42
mgoddardif someone was keen to get going15:42
mgoddardAnother one is the support matrix definition15:42
*** irclogbot_3 has quit IRC15:42
mgoddardWe need someone to go through all the services & features we support, and help define a level of 'support' for them15:43
*** henriqueof has joined #openstack-kolla15:43
mgoddardI started listing images15:43
mgoddard#link https://etherpad.openstack.org/p/kolla-train-image-evaluation15:43
mgoddardsome projects we 'support' are dead :)15:43
mgoddardsome were for k8s, we can drop those15:44
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: ceph-nfs: Add rpcbind to Ubuntu host bootstrap  https://review.opendev.org/66931515:44
mgoddardAnyone want to help with that?15:44
yoctozeptomgoddard: image list != feature list15:44
mgoddardyoctozepto: nope, that's just a start15:44
yoctozepto;-)15:44
mnasiadkayoctozepto: not all images are OpenStack project related :)15:45
mgoddardfeatures probably more applicable to k-a15:45
yoctozeptoyeah, thought about k-a15:45
mnasiadkaso let's start with kolla - easier?15:45
mgoddardyeah15:45
yoctozeptoyeah15:45
*** altlogbot_1 has joined #openstack-kolla15:45
mgoddardI found some low hanging fruit - images we can drop straight away15:45
*** cgrosjean has quit IRC15:46
openstackgerritGaëtan Trellu proposed openstack/kolla master: Add HAcluster containers  https://review.opendev.org/66876515:46
yoctozeptomgoddard: yeah, mariadb15:46
mnasiadkamgoddard: a lot of kolla-kubernetes remnants I assume15:46
mgoddardAnother feature without an owner is health checks15:46
mgoddardyoctozepto: yeah mariadb will be dropped in train15:46
mgoddard:p15:46
mnasiadkamgoddard: I can take this, had a preliminary look into how tripleo does that, shouldn't be a lot of work15:47
mgoddardAnd always looking to improve test coverage, so if there's a service you use that is untested, please add a test (we can help with this)15:47
*** henriqueof has quit IRC15:47
mgoddardmnasiadka: that would be good15:47
*** altlogbot_1 has quit IRC15:48
mgoddardthey're the main ones - please see the full list if you are interested in picking up a feature, and we can help explain if you need context15:48
mgoddardanything else for train planning?15:49
*** gchenuet has joined #openstack-kolla15:49
mgoddard#topic open discussion15:49
*** openstack changes topic to "open discussion (Meeting topic: kolla)"15:49
gchenuetHi guys ! Congrats for your work on Kolla and Kolla-ansible. Is there a relase date planned for kolla/kolla-ansible 8.0.0.0 ?15:49
goldyfruitplop Guillaume :_15:50
goldyfruito/15:50
goldyfruitgchenuet,15:50
mgoddardgchenuet: hi, we're hoping for next week15:50
mgoddardbut it depends on some ongoing testing of mariadb15:50
mgoddard(see earlier discussion)15:50
mgoddardWe had a bit of a discussion on monday in the (last ever) kayobe meeting about how to integrate15:50
gchenuetgoldyfruit: \o15:51
mgoddard#link http://eavesdrop.openstack.org/meetings/kayobe/2019/kayobe.2019-07-08-14.01.log.html15:51
gchenuetOh cool ! Good news :)15:51
mgoddardwe thought we should combine the kayobe whiteboard into the kolla one15:51
mgoddard#link https://etherpad.openstack.org/p/kayobe-whiteboard15:51
mgoddarddoes that make sense?15:51
mgoddardkeeps things in one place15:51
mgoddardwe'll also be closing the #openstack-kayobe channel and moving in here, so expect more kayobe chatter15:52
mgoddardthe gerritbot has already been updated to push notifications here15:52
mgoddardI'll introduce the kayobe team next week's meeting15:53
mgoddardbut I'm sure you'll have seen them all around15:53
mnasiadka More people complaining on MariaDB - the merrier :)15:54
mgoddardthat's mostly what we do yeah15:54
mgoddardAnything else to discuss?15:55
*** altlogbot_0 has joined #openstack-kolla15:55
mgoddardOk, thanks everyone15:57
mgoddard#endmeeting15:57
*** openstack changes topic to "Topic for #openstack-kolla is: support: ask.openstack.org | New to Kolla: docs @ https://docs.openstack.org/kolla/latest/ | | Kolla IRC meetngs on Wednesdays @ 15:00 UTC - see agenda @ https://goo.gl/OXB0DL - IRC channel is *LOGGED* @ http://goo.gl/3mzZ7b"15:57
openstackMeeting ended Wed Jul 10 15:57:36 2019 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)15:57
openstackMinutes:        http://eavesdrop.openstack.org/meetings/kolla/2019/kolla.2019-07-10-15.00.html15:57
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/kolla/2019/kolla.2019-07-10-15.00.txt15:57
openstackLog:            http://eavesdrop.openstack.org/meetings/kolla/2019/kolla.2019-07-10-15.00.log.html15:57
*** altlogbot_0 has quit IRC15:58
ohwhyosaThe list of images/services, what help do you need? Checking their supported status on other openstack services? or in general if their repos are active, in case they are not core openstack?15:59
mgoddardohwhyosa: it's more about us - we agreed on 3 levels of 'support'15:59
mgoddardsupported/maintained/community supported16:00
mgoddardso core things are supported - we will try hard to fix when (badly) broken16:00
mgoddardmaintained needs definition, but probably these would be other things that we test in CI16:00
mgoddardcommunity maintained is best effort - we accept patches and may try to fix ourselves, but do not commit16:01
mgoddardfor these we may look for 'owners', e.g. goldyfruit for masakari16:01
goldyfruitmgoddard, I'm in to maintain Qinling, Masakari and HAcluster (if merged)16:02
mgoddardgoldyfruit: great16:02
mgoddardohwhyosa: part of the work is about definiting these terms, and what they mean16:03
mgoddardthe other part is about applying them to the various services and features for kolla and kolla ansible16:03
mgoddardmake sense?16:03
ohwhyosaYeah, makes sense!16:05
ohwhyosaSo the first part I guess is more of an internal/core effort, but if you need help with latter, be glad to help in some way or another16:07
mnasiadkamgoddard: https://review.opendev.org/#/c/669315/ - we need your opinion :)16:09
*** Wasaac has quit IRC16:10
goldyfruitmgoddard, by the way, congrats on the baby!16:11
*** gtarnaras has quit IRC16:11
yoctozeptomgoddard: boy/girl? (or rude to ask?)16:14
mgoddardgoldyfruit: thanks16:16
mgoddardyoctozepto: boy16:16
*** michaelbarkdoll has joined #openstack-kolla16:17
yoctozeptomgoddard: like father like son16:17
yoctozeptowill be fixing mariadb in no time ;-)16:17
goldyfruit^^16:17
mgoddardyoctozepto: that is the plan, then I can retire16:17
yoctozeptomgoddard: I guess it would be evil to call your hypothetical future daughter Maria ;D16:19
goldyfruitxD16:20
mgoddardmnasiadka: commented16:20
mgoddardyoctozepto: especially since her brother is called galera16:20
yoctozeptomgoddard: crying laughing16:21
*** henriqueof has joined #openstack-kolla16:21
yoctozeptothat was too much16:21
goldyfruitThey will have inseparable like, HA16:21
goldyfruitthey will be*16:21
yoctozeptogoldyfruit: though I wonder who will be the slave16:21
mgoddardhopefully they will be more reliable than galera16:21
goldyfruityoctozepto, always the youngest16:21
goldyfruitmgoddard, meaning you need a third one16:22
goldyfruitin less than 10 minutes, you got 3 children working in IT \o/16:23
yoctozeptomgoddard: what name did you choose really?16:24
mgoddardstill deciding16:24
yoctozeptomgoddard: yeah, you still have some time16:24
mgoddardquorum difficult with two members :)16:24
goldyfruit:D16:24
mgoddardother member not keen on galera16:25
goldyfruitshort list ?16:25
yoctozeptoyeah, in this light love triangles make much more sense16:25
michaelbarkdollDoes external ceph need gnocchi and manila?  If so, do I need to create osd pools for each of them with what level of access?  Would something like `ceph auth get-or-create client.gnocchi mon 'profile rbd' osd 'profile rbd pool=gnocchi'` be required or do they just use the volumes, images, backups, vms pools?16:26
yoctozeptomichaelbarkdoll: no16:26
yoctozeptothey are supported, not required16:26
michaelbarkdollThanks16:26
yoctozeptoyou also may opt not to use cinder16:27
yoctozeptoor glance16:27
yoctozeptoor nova16:27
*** k_mouza has quit IRC16:27
yoctozeptomix and match16:27
yoctozepto;D16:27
yoctozeptominimal sensible setup is nova and glance connected to ceph though16:27
yoctozeptowell, about mariadb issues, mgoddard had a point in creating a DNM change for many exact jobs doing multinode mariadb16:28
yoctozeptopreferably without ceph16:29
yoctozeptoto not look at those other failures16:29
*** henriqueof has quit IRC16:29
yoctozeptoI can then grind some more thought material16:29
yoctozeptofor now I want to finish some changes I have in making16:32
*** Sravan has joined #openstack-kolla16:32
*** quantomworks has quit IRC16:36
*** henriqueof has joined #openstack-kolla16:37
goldyfruityoctozepto, about you comment in https://review.opendev.org/#/c/668765/16:40
goldyfruitWhat do you mean by "no ubuntu" ?16:40
goldyfruitThe images I'm using to test are built on Bionic16:40
*** factor has joined #openstack-kolla16:41
goldyfruithttps://paste.api-zulu.com/aqafabiyed.go16:43
yoctozeptogoldyfruit: sorry, I saw only rpm customizations in there and thought you missed ubuntu16:43
yoctozeptoso it's that centos just needs extra repos while ubuntu not16:43
yoctozeptoand packages in both share the same names?16:44
yoctozeptothat's rare ;D16:44
goldyfruitthats old \o/16:44
goldyfruitBut yes, crmsh is in debian/ubuntu16:44
*** hamzaachi has quit IRC16:47
openstackgerritMerged openstack/kolla-ansible master: Update designate-guide cli command for dns_domain  https://review.opendev.org/67002416:48
*** hamzaachi has joined #openstack-kolla16:48
*** altlogbot_1 has joined #openstack-kolla16:49
openstackgerritMark Goddard proposed openstack/kolla-ansible master: init-runonce: make public1 network optional  https://review.opendev.org/67012816:50
openstackgerritMerged openstack/kolla-ansible master: Fix the incorrect backup_driver configuration  https://review.opendev.org/67000616:50
openstackgerritMark Goddard proposed openstack/kolla-ansible stable/stein: init-runonce: make public1 network optional  https://review.opendev.org/67012916:51
*** altlogbot_1 has quit IRC16:54
openstackgerritMark Goddard proposed x/kayobe master: CI: Don't create an external network in init-runonce  https://review.opendev.org/67013116:54
*** Sravan has quit IRC16:57
*** hamzaachi has quit IRC16:58
*** henriqueof has quit IRC17:03
*** dougsz has quit IRC17:03
*** henriqueof has joined #openstack-kolla17:03
michaelbarkdollIs there a preferred topology for running ceph-ansible with openstack-kolla nodes? Reason that I ask is that I have an error occurring during kolla-ansible bootstrap process after ceph-ansible is deployed to some nodes in the cluster and I try to also define the ceph-ansible nodes as storage nodes in the kolla-ansible inventory file.  The error is17:05
michaelbarkdollabout a docker package version conflict (docker-ce conflicts with 2:docker-1.13.1-96) that was installed by ceph-ansible on my storage nodes ceph-ansible cluster and the version that kolla-ansible's bootstrap playbook wants to install.  I'm curious what is the best approach to using ceph-ansible storage nodes with kolla-ansible nodes.  Is it best p17:05
michaelbarkdollractice to just keep them separate and not define them in the kolla-ansible inventory file or should ceph-ansible be made to run in containers to avoid the version conflict?17:05
yoctozeptomichaelbarkdoll: interesting point17:05
yoctozeptothe latest docker is the best usually17:06
yoctozeptowe need instructions written for this integration for train17:06
*** Sravan has joined #openstack-kolla17:06
yoctozeptopreferably deploy latest docker before deploying either part17:07
yoctozepto(ceph/kolla)17:07
michaelbarkdollOk, I'll give that a try, thanks17:07
*** Sravan has quit IRC17:09
*** Sravan has joined #openstack-kolla17:09
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: Add neutron precheck for tunnel interface and its address  https://review.opendev.org/67013817:12
kplantmichaelbarkdoll: to resolve that i deployed ceph via ceph-ansible on metal17:13
kplantand just use python-venvs17:13
*** Sravan has quit IRC17:15
yoctozeptokplant, michaelbarkdoll: and I have ceph deployed on dedicated metal without using ceph-ansible at all17:15
kplantshun!17:15
kplantshun the ceph-deploy users17:15
yoctozeptowithout ceph-deploy either17:15
kplantdo tell17:16
michaelbarkdollI'm currently using VMs for the ceph-ansible deployment (similar to metal), but I thought python-venvs were just for python pip installed packages isolation.  Does it somehow allow for different versions of docker as well?17:16
kplantmichaelbarkdoll: if you deploy non-containerized ceph you don't need to worry about docker17:16
michaelbarkdollOh odd that I had a version conflict then, hmm17:16
michaelbarkdollYeah, I must have something wrong, need to investigate it more.17:17
kplantyou might have a docker: yes in all.yml17:17
kplantif you really really want containerized ceph, you can modify one of the roles to just not install docker17:17
kplantthen it will use the docker installed by kolla-ansible bootstrap-servers17:18
yoctozeptoor maybe you installed docker for yerself ;D17:18
*** Sravan has joined #openstack-kolla17:18
michaelbarkdollWell, it isn't in all.yml so I'm guessing I did something stupid somewhere...17:19
michaelbarkdollNice thing about vagrant is it won't take long to find it :)17:20
*** rpittau is now known as rpittau|afk17:23
*** rouk has joined #openstack-kolla17:24
*** Sravan has quit IRC17:24
*** Sravan has joined #openstack-kolla17:26
*** Sravan has quit IRC17:34
*** henriqueof has quit IRC17:34
*** Sravan has joined #openstack-kolla17:35
*** henriqueof has joined #openstack-kolla17:35
*** Sravan has quit IRC17:36
*** Sravan has joined #openstack-kolla17:36
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible stable/stein: Fix the incorrect backup_driver configuration  https://review.opendev.org/67014317:39
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible stable/stein: Update designate-guide cli command for dns_domain  https://review.opendev.org/67014417:40
*** BjoernT has quit IRC17:41
*** dpawlik has joined #openstack-kolla17:48
*** gchenuet has quit IRC17:51
openstackgerritMerged openstack/kolla stable/rocky: Write hash after compressing horizon static assets  https://review.opendev.org/66980517:57
*** irclogbot_0 has joined #openstack-kolla18:05
*** irclogbot_0 has quit IRC18:08
*** heikkine has quit IRC18:38
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: Enhance merge_* action plugins to allow expected relative includes  https://review.opendev.org/67015818:43
henriqueofCan I use ODL installed with kolla-ansible as (BGP) route server for public networks?18:53
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: Enhance merge_* action plugins to allow expected relative includes  https://review.opendev.org/67015818:53
yoctozeptohenriqueof: if you are asking for our permission then hereby I grant you mine ;D18:54
henriqueofyoctozepto: haha, actually I was asking if it is possible.18:54
yoctozeptothough BGP being generally fragile I doubt you want it this way, can you explain further?18:54
yoctozeptowe don't disallow it specifically18:55
henriqueofyoctozepto: I want to make an IBGP between ODL and my BGP server and advertise routes for the public addresses used as floating IPs.18:56
yoctozeptohenriqueof: is it geographically relevant? if not, I would just inject whole subnets into BGP18:58
kplantis there a reason you're not just advertising the whole cidr?18:58
yoctozeptokplant: 4 secs late ;p18:59
*** priteau has quit IRC18:59
kplanti'm on the moon so there's a lot of latency18:59
yoctozeptokplant: I see18:59
kplant:D18:59
kplantfrom a geographic standpoint that would make sense with eBGP but then you'd have to worry about neutron not allocating the same fip in two places19:00
henriqueofyoctozepto: Actually I am doing it right now, advertise whole cidr and make static routes from my PE.19:00
henriqueofI am doing it to balance routing load from the main BGP router.19:01
*** altlogbot_0 has joined #openstack-kolla19:01
yoctozeptokplant: unless henriqueof wants anycast ;-)19:02
henriqueofThis way my BGP router would only forward packets and the routing will be done by ODL/neutron.19:02
henriqueofActually it is an initial idea, still thinking about it. ;)19:02
yoctozeptohenriqueof: but it's a matter of a static route then19:02
yoctozeptoI don't see where iBGP would help here19:03
*** altlogbot_0 has quit IRC19:04
henriqueofHmmm.19:04
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: Do not require valid migration_interface for controllers  https://review.opendev.org/66963119:05
*** KeithMnemonic has joined #openstack-kolla19:10
*** BjoernT has joined #openstack-kolla19:11
henriqueofWhat is the relation between 'enable_neutron_lbaas' and 'enable_octavia'? Do they depend on each other, or just enable_octavia should work?19:13
goldyfruitenable_octavia should work19:18
*** ivve has joined #openstack-kolla19:18
goldyfruito/ ivve19:18
goldyfruithenriqueof, https://docs.openstack.org/releasenotes/kolla-ansible/unreleased.html19:19
*** irclogbot_2 has joined #openstack-kolla19:19
goldyfruitCheck the deprecation note19:19
henriqueofgoldyfruit: Thanks!19:20
*** irclogbot_2 has quit IRC19:22
*** whoami-rajat has quit IRC19:24
michaelbarkdollCould ceph-ansible stable-4.0 branch's dashboard maybe be using docker?  Docker only seems to be installed on ceph nodes after the deploy step of ceph-ansible. Here is my ceph-ansible's all.yml http://paste.openstack.org/show/754275/  Seems task, "ceph-container-engine : install container package" is being run for some reason.  kplant what version19:26
michaelbarkdollof ceph-ansible did you use to deploy?19:26
kplantstable-3.2 and now stable-4.019:30
kplantno problems with either19:30
kplanti would suggest copying all.yml.sample and editing that19:30
michaelbarkdollOk, reloading my stack going to try it without the dashboard maybe it uses docker for some reason19:30
kplantthere's a ton of switches19:30
michaelbarkdollSure, I can try that too19:30
*** Sravan has quit IRC19:30
openstackgerritGaëtan Trellu proposed openstack/kolla-ansible master: WIP: Add HAcluster Ansible role  https://review.opendev.org/67010419:32
openstackgerritGaëtan Trellu proposed openstack/kolla-ansible master: WIP: Add HAcluster Ansible role  https://review.opendev.org/67010419:36
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: DNM: Test MariaDB failures in CI  https://review.opendev.org/67017019:38
goldyfruitDo you know where can I find the logs of the CI19:39
goldyfruitcrm configure property stonith-enabled=false19:39
goldyfruitoups19:39
goldyfruit2019-07-10 14:19:05.307599 | TASK [Run deploy.sh script]19:39
goldyfruit2019-07-10 14:43:44.505180 | primary | ERROR19:39
goldyfruit2019-07-10 14:43:44.551197 | primary | {19:39
goldyfruit2019-07-10 14:43:44.552661 | primary |   "msg": "non-zero return code",19:39
goldyfruit2019-07-10 14:43:44.552784 | primary |   "rc": 119:39
goldyfruit2019-07-10 14:43:44.553267 | primary | }19:39
goldyfruitI would like to see what fails in deploy.sh19:39
yoctozeptogoldyfruit: primary/logs/ansible/deploy19:39
yoctozeptoor even primary/ara19:39
goldyfruitcool yoctozepto thx !19:40
yoctozeptoyw19:40
goldyfruitImageNotFound: 404 Client Error: Not Found (\"pull access denied for kolla/ubuntu-source-masakari-api19:41
goldyfruitI guess I need to add the containers review19:42
goldyfruitas depends-on19:42
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: DNM: Test MariaDB failures in CI  https://review.opendev.org/67017019:43
yoctozeptogoldyfruit: definitely19:43
goldyfruityoctozepto, should I add a new depends-on line ?19:44
goldyfruitOr should I add the other review on the same line ?19:44
yoctozeptogoldyfruit: another19:45
goldyfruitThx!19:45
openstackgerritGaëtan Trellu proposed openstack/kolla-ansible master: Testing Masakari role in gate  https://review.opendev.org/61605019:46
*** openstackgerrit has quit IRC19:49
*** kplant has quit IRC19:54
*** factor has quit IRC19:55
*** Sravan has joined #openstack-kolla20:01
henriqueofI just tried kolla-ansible branch stable/stein on a Ubuntu Server 18.04 box, it passes the install packages step but says no pip3 installed, shouldn't it be done automatically?20:02
*** Sravan has quit IRC20:06
*** factor has joined #openstack-kolla20:06
yoctozeptohenriqueof: report bug as detailed as possible20:07
yoctozeptoon our launchpad20:07
yoctozeptogood night kolla channel20:07
goldyfruityoctozepto, see ya o/20:07
henriqueofyoctozepto: Okay, good night!20:08
henriqueofI installed pip3 manually on the server and stepped on another problem. lol20:08
henriqueofOn the 'Get current kolla-ansible version number' task it fails with 'Exception: Versioning for this project requires either an sdist tarball, or access to an upstream git repository'20:11
*** ivve has quit IRC20:14
*** openstackgerrit has joined #openstack-kolla20:18
openstackgerritGaëtan Trellu proposed openstack/kolla-ansible master: Testing Masakari role in gate  https://review.opendev.org/61605020:18
goldyfruithenriqueof, If you run this command: python -c "import pbr.version; print(pbr.version.VersionInfo('kolla-ansible'))"20:19
henriqueofgoldyfruit: Nevermind it was my bad, since I wasn't able to clone github repos earlier I downloaded the zip file, that does not contain repo info.20:21
henriqueofgoldyfruit: Mande a clone and it seems to be working.20:21
goldyfruitok20:21
*** dpawlik has quit IRC20:27
openstackgerritGaëtan Trellu proposed openstack/kolla-ansible master: Testing Masakari role in gate  https://review.opendev.org/61605020:27
*** altlogbot_1 has joined #openstack-kolla20:27
*** altlogbot_1 has quit IRC20:32
*** Sravan has joined #openstack-kolla20:34
*** Sravan has quit IRC20:39
*** dpawlik has joined #openstack-kolla20:44
*** pcaruana has quit IRC20:48
*** dpawlik has quit IRC20:48
openstackgerritGaëtan Trellu proposed openstack/kolla-ansible master: WIP: Add HAcluster Ansible role  https://review.opendev.org/67010420:56
*** cloudnull has quit IRC21:01
*** cloudnull has joined #openstack-kolla21:02
*** altlogbot_3 has joined #openstack-kolla21:03
*** altlogbot_3 has quit IRC21:08
*** altlogbot_1 has joined #openstack-kolla21:17
*** altlogbot_1 has quit IRC21:22
*** Sravan has joined #openstack-kolla21:24
*** Sravan has quit IRC21:29
*** Sravan has joined #openstack-kolla21:31
*** factor has quit IRC21:45
*** factor has joined #openstack-kolla21:46
*** irclogbot_3 has joined #openstack-kolla21:47
*** goldyfruit has quit IRC21:47
*** irclogbot_3 has quit IRC21:52
*** kplant has joined #openstack-kolla21:56
*** henriqueof has quit IRC22:06
*** mchlumsky has quit IRC22:10
*** luksky11 has quit IRC22:43
*** altlogbot_0 has joined #openstack-kolla23:03
*** altlogbot_0 has quit IRC23:08
*** altlogbot_1 has joined #openstack-kolla23:13
*** altlogbot_1 has quit IRC23:18
*** altlogbot_0 has joined #openstack-kolla23:19
*** altlogbot_0 has quit IRC23:24
*** altlogbot_0 has joined #openstack-kolla23:25
*** altlogbot_0 has quit IRC23:30
*** Sravan has quit IRC23:32
*** zhanglong has joined #openstack-kolla23:44
*** zhanglong has quit IRC23:46
*** zhanglong has joined #openstack-kolla23:54

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!