Monday, 2016-06-27

*** v1k0d3n has joined #openstack-kolla00:13
*** Mr_Broken has joined #openstack-kolla00:38
*** Mr_Broken has quit IRC00:42
*** daneyon has joined #openstack-kolla00:45
*** daneyon has quit IRC00:50
*** dwalsh has quit IRC00:51
openstackgerritJack Ning proposed openstack/kolla: Replace horizon default config with custom config  https://review.openstack.org/30692800:55
*** zhurong has joined #openstack-kolla01:16
*** salv-orl_ has quit IRC01:25
*** phuongnh has joined #openstack-kolla01:29
*** g3ek has quit IRC01:40
*** haplo37 has quit IRC01:40
*** zhiwei has joined #openstack-kolla01:45
*** haplo37 has joined #openstack-kolla01:47
*** g3ek has joined #openstack-kolla01:48
*** JianqingJiang has joined #openstack-kolla01:51
*** salv-orlando has joined #openstack-kolla01:56
*** hanchao has joined #openstack-kolla01:56
openstackgerritBritt Houser proposed openstack/kolla: Fix minor typo in security guide.  https://review.openstack.org/33427202:04
*** salv-orlando has quit IRC02:05
openstackgerritBritt Houser proposed openstack/kolla: [TrivialFix] Typo in security guide.  https://review.openstack.org/33427202:23
*** sdake_ has quit IRC02:23
*** Mr_Broken has joined #openstack-kolla02:26
*** Mr_Broken has quit IRC02:31
openstackgerritBritt Houser proposed openstack/kolla: Fix minor typo in security guide.  https://review.openstack.org/33427202:32
*** yuanying has quit IRC02:49
*** klint has joined #openstack-kolla02:49
*** salv-orlando has joined #openstack-kolla03:02
openstackgerritMd Nadeem proposed openstack/kolla: Fix container stop exeption  https://review.openstack.org/33374403:03
*** salv-orlando has quit IRC03:10
openstackgerritMerged openstack/kolla: Fix minor typo in security guide.  https://review.openstack.org/33427203:12
openstackgerritMerged openstack/kolla: glance-*, fix rabbit config with ceilometer  https://review.openstack.org/33420503:13
openstackgerritMerged openstack/kolla: Fix docker daemon proxy support in vagrant  https://review.openstack.org/33323803:17
*** coolsvap has joined #openstack-kolla03:17
*** Mr_Broken has joined #openstack-kolla03:20
*** v1k0d3n has quit IRC03:20
*** Jeffrey4l_ has quit IRC03:23
*** robcresswell has quit IRC03:24
*** Mr_Broken has quit IRC03:25
*** Daviey has quit IRC03:25
*** Daviey has joined #openstack-kolla03:27
*** robcresswell has joined #openstack-kolla03:27
*** Jeffrey4l_ has joined #openstack-kolla03:36
*** sacharya has joined #openstack-kolla03:41
*** yuanying has joined #openstack-kolla03:47
openstackgerritHui Kang proposed openstack/kolla: Enable openvswitch container logs in host volumes  https://review.openstack.org/33428303:49
mdnadeemcoolsvap, ping03:59
*** Jeffrey4l_ has quit IRC04:00
hanchaohi guys, have you suffered with the unstable rabbitmq image for Liberty(1.1.0)?04:01
*** stvnoyes has quit IRC04:01
*** stvnoyes has joined #openstack-kolla04:02
mdnadeemhanchao, No idea for v 1.1.0, however i have build it on Liberrty(1.1.1) Centos successfully04:04
hanchaothe image itself can be built successfully. but when i deploy that, it's really unstable... don't know what's your experience?04:06
*** salv-orlando has joined #openstack-kolla04:07
mdnadeemhanchao, i have deployedon v 1.1.1 successfully04:08
*** Jeffrey4l_ has joined #openstack-kolla04:09
hanchaoand your openstack cluster works well?04:09
hanchaoif so, maybe i should try that.04:09
*** gbraad has joined #openstack-kolla04:11
mdnadeemhanchao, yes, thats work fine04:12
hanchaomdnadeem: btw, and did you built all images by this tag? or only for the rabbitmq?04:13
mdnadeemhanchao, all image04:13
hanchaoI'll try it, thx :)04:15
*** salv-orlando has quit IRC04:15
coolsvapmdnadeem, pong whatsup?04:21
*** Jeffrey4l_ has quit IRC04:21
mdnadeemcoolsvap, hi04:24
mdnadeemcoolsvap, my haproxy container restart because of nova-novnc proxy, haproxy tries to bind socket [192.168.122.60:6080]04:24
hanchaoI proposed a suggestion if anyone could have a look at and leave comments [https://review.openstack.org/#/c/333666/ ], thx in advance.04:25
patchbothanchao: patch 333666 - kolla - Add the verification of required images step befor...04:25
mdnadeemhowever, it haproxy container log show : Running command: '/usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid'04:25
mdnadeem[WARNING] 178/031940 (8) : config : 'option forwardfor' ignored for proxy 'mariadb' as it requires HTTP mode.04:25
mdnadeem[ALERT] 178/031940 (8) : Starting proxy nova_novncproxy: cannot bind socket [192.168.122.60:6080]04:25
mdnadeemcoolsvap, nova-novnc service already bind this port to 0.0.0.0, although in nova.conf novncproxy_host = 192.168.122.185 is defined04:27
mdnadeemI am worried why nova-novncproxy service bind to 0.0.0.0 even when novncproxy_host is defined in nova.conf ?04:27
coolsvapmdnadeem, i dont have a multinode setup to test and verify tbh but i will give a try04:28
mdnadeemcoolsvap, ohk, thanks04:29
*** haplo37 has quit IRC04:34
*** Jeffrey4l_ has joined #openstack-kolla04:39
*** salv-orlando has joined #openstack-kolla04:42
*** tfukushima has joined #openstack-kolla04:42
*** haplo37 has joined #openstack-kolla04:42
openstackgerritMerged openstack/kolla: Fix container stop exeption  https://review.openstack.org/33374404:45
*** salv-orlando has quit IRC04:50
*** daneyon has joined #openstack-kolla05:16
*** tfukushima has quit IRC05:19
*** g3ek has quit IRC05:20
*** daneyon has quit IRC05:21
*** haplo37 has quit IRC05:21
*** g3ek has joined #openstack-kolla05:26
*** haplo37 has joined #openstack-kolla05:27
*** salv-orlando has joined #openstack-kolla05:38
*** fragatina has joined #openstack-kolla05:40
*** fragatina has quit IRC05:40
*** fragatina has joined #openstack-kolla05:41
*** tfukushima has joined #openstack-kolla05:49
*** JeroenBo has joined #openstack-kolla05:59
*** diga has joined #openstack-kolla06:02
*** haplo37 has quit IRC06:02
*** Mr_Broken has joined #openstack-kolla06:02
*** g3ek has quit IRC06:04
*** g3ek has joined #openstack-kolla06:04
*** haplo37 has joined #openstack-kolla06:05
*** bootsha has joined #openstack-kolla06:05
*** Mr_Broken has quit IRC06:07
*** neilus has joined #openstack-kolla06:19
*** v1k0d3n has joined #openstack-kolla06:35
*** v1k0d3n has quit IRC06:40
*** g3ek has quit IRC06:42
*** haplo37 has quit IRC06:46
openstackgerritMerged openstack/kolla: Stop using a global logger for all the things  https://review.openstack.org/32188406:48
*** haplo37 has joined #openstack-kolla06:48
openstackgerritMohammed Salih Puthenpurayil proposed openstack/kolla: Trivial fixes to correct couple of typos.  https://review.openstack.org/33430606:50
*** g3ek has joined #openstack-kolla06:51
*** Mr_Broken has joined #openstack-kolla06:53
*** Serlex has joined #openstack-kolla06:54
*** sacharya has quit IRC06:55
*** Mr_Broken has quit IRC07:05
*** Mr_Broken has joined #openstack-kolla07:08
*** belmoreira has joined #openstack-kolla07:09
*** Mr_Broke_ has joined #openstack-kolla07:21
*** Mr_Broken has quit IRC07:23
*** Mr_Broken has joined #openstack-kolla07:27
*** Mr_Broke_ has quit IRC07:29
*** dmk0202 has joined #openstack-kolla07:31
*** Mr_Broken has quit IRC07:31
*** Mr_Broken has joined #openstack-kolla07:31
*** Mr_Broken has quit IRC07:37
*** Mr_Broken has joined #openstack-kolla07:41
*** jmccarthy has left #openstack-kolla07:42
*** shardy has joined #openstack-kolla07:47
*** Mr_Broken has quit IRC07:53
*** sacharya has joined #openstack-kolla07:56
*** Mr_Broken has joined #openstack-kolla07:58
*** b_bezak has joined #openstack-kolla08:00
*** Mr_Broke_ has joined #openstack-kolla08:01
*** sacharya has quit IRC08:01
*** athomas has joined #openstack-kolla08:03
*** Mr_Brok__ has joined #openstack-kolla08:03
*** neilus has quit IRC08:03
*** Mr_Broken has quit IRC08:03
*** salv-orlando has quit IRC08:04
*** Mr_Broke_ has quit IRC08:06
*** neilus has joined #openstack-kolla08:06
*** Mr_Broken has joined #openstack-kolla08:08
*** Mr_Brok__ has quit IRC08:09
*** Mr_Broke_ has joined #openstack-kolla08:11
*** mewald has joined #openstack-kolla08:13
*** Mr_Broken has quit IRC08:13
*** g3ek has quit IRC08:15
gbraad.join #bifrost08:15
*** haplo37 has quit IRC08:16
*** gbraad has quit IRC08:16
*** gbraad has joined #openstack-kolla08:16
*** Mr_Broken has joined #openstack-kolla08:19
*** Mr_Broke_ has quit IRC08:21
*** haplo37 has joined #openstack-kolla08:21
*** g3ek has joined #openstack-kolla08:21
*** Mr_Broken has quit IRC08:23
*** salv-orlando has joined #openstack-kolla08:25
*** dwalsh has joined #openstack-kolla08:27
*** shardy has quit IRC08:32
*** shardy has joined #openstack-kolla08:34
*** Mr_Broken has joined #openstack-kolla08:34
*** phuongnh has quit IRC08:34
*** Mr_Broke_ has joined #openstack-kolla08:38
*** Mr_Broken has quit IRC08:39
openstackgerritMd Nadeem proposed openstack/kolla: Add unit test for stop and restart container  https://review.openstack.org/33152408:42
*** Mr_Broken has joined #openstack-kolla08:45
*** Mr_Broke_ has quit IRC08:46
*** Mr_Broken has quit IRC08:50
*** Mr_Broken has joined #openstack-kolla08:51
*** dwalsh has quit IRC08:51
*** tfukushima has quit IRC08:52
*** tfukushima has joined #openstack-kolla08:53
*** sacharya has joined #openstack-kolla08:57
*** Mr_Broken has quit IRC08:58
*** g3ek has quit IRC08:59
*** Mr_Broken has joined #openstack-kolla08:59
*** haplo37 has quit IRC09:00
*** neilus has quit IRC09:00
*** haplo37 has joined #openstack-kolla09:01
*** neilus has joined #openstack-kolla09:01
*** g3ek has joined #openstack-kolla09:01
*** sacharya has quit IRC09:01
openstackgerritMohammed Salih Puthenpurayil proposed openstack/kolla: Trivial fixes to correct couple of typos.  https://review.openstack.org/33430609:03
JianqingJiangCan Kolla achieve high availability?09:05
*** Mr_Broke_ has joined #openstack-kolla09:07
*** Mr_Broken has quit IRC09:09
sean-k-mooneykolla supprots ha deployment of most if not all the services it can deploy09:11
sean-k-mooneyachiveing high availablity is rathaer subjective though09:11
*** Mr_Broken has joined #openstack-kolla09:14
*** Mr_Broke_ has quit IRC09:16
*** Mr_Broke_ has joined #openstack-kolla09:16
*** Mr_Broken has quit IRC09:18
*** Mr_Broken has joined #openstack-kolla09:18
JianqingJiangOK.But I think if there is a config file to customize HA option is better09:20
*** Mr_Broke_ has quit IRC09:20
*** Mr_Broken has quit IRC09:23
*** Mr_Broken has joined #openstack-kolla09:28
*** Mr_Broke_ has joined #openstack-kolla09:31
*** Mr_Broken has quit IRC09:32
*** Mr_Broken has joined #openstack-kolla09:33
*** Mr_Broke_ has quit IRC09:35
*** Mr_Broken has quit IRC09:37
*** Mr_Broken has joined #openstack-kolla09:38
sean-k-mooneyJianqingJiang: you can customise the ha deployment via the ansible inventory file.09:40
*** g3ek has quit IRC09:43
*** Mr_Broke_ has joined #openstack-kolla09:43
JianqingJiangIs there some documents about HA deployment via ansible inventory file?09:44
*** prithiv has joined #openstack-kolla09:44
pbourkeJianqingJiang: http://docs.openstack.org/developer/kolla/multinode.html09:44
*** haplo37 has quit IRC09:44
sean-k-mooneykolla is ha by defult if you are not using singel node09:45
*** Mr_Broken has quit IRC09:45
*** haplo37 has joined #openstack-kolla09:45
*** zhiwei has quit IRC09:46
*** g3ek has joined #openstack-kolla09:46
*** bootsha has quit IRC09:47
*** neilus has quit IRC09:47
*** neilus has joined #openstack-kolla09:48
*** neilus has quit IRC09:48
*** neilus has joined #openstack-kolla09:49
*** neilus has quit IRC09:49
*** neilus has joined #openstack-kolla09:50
pbourkecorrect09:50
*** mewald has quit IRC09:51
*** tyrola has joined #openstack-kolla09:52
JianqingJiangBut I notice that every service is only one container to provide09:52
*** prithiv has quit IRC09:55
*** Mr_Broken has joined #openstack-kolla09:55
*** strigazi_ is now known as strigazi09:56
*** Mr_Broke_ has quit IRC09:57
*** zhurong has quit IRC10:00
*** Mr_Broke_ has joined #openstack-kolla10:02
sean-k-mooneyJianqingJiang: can you expand on that?10:02
sean-k-mooneyJianqingJiang: each container runs a singel service. ha deployment are achived by running10:02
*** Mr_Broken has quit IRC10:02
sean-k-mooneyJianqingJiang: several instances of the same container on multiple hosts10:03
sean-k-mooneyJianqingJiang: haporxy and keepalived are then used to loadblance across the api services10:03
sean-k-mooneyJianqingJiang: mariadb is clusered using glara if i recall and i think Rabbit MQs native clustering is also used.10:04
JianqingJiangThat is for stateless service,but how to achieve ha about stateful service?10:04
JianqingJiangOK10:04
JianqingJiangThanks a lot10:05
*** neilus has quit IRC10:07
*** Mr_Broken has joined #openstack-kolla10:08
*** Mr_Broke_ has quit IRC10:09
*** b_bezak has quit IRC10:12
*** b_bezak has joined #openstack-kolla10:13
*** mewald has joined #openstack-kolla10:13
*** Mr_Broke_ has joined #openstack-kolla10:13
*** Mr_Broken has quit IRC10:14
*** dims has quit IRC10:14
*** Mr_Broken has joined #openstack-kolla10:16
*** Mr_Broke_ has quit IRC10:19
*** dims has joined #openstack-kolla10:20
*** JianqingJiang has quit IRC10:24
*** Mr_Broken has quit IRC10:24
tyrolaHey guys, I am fighting with neutron openvswitch in our kolla setup now for 2 days. I've running a multinode cluster with kolla on cent os 7. But the router gateway address and the floating ips are not reachable. I can't find the reason. I also see no errors in the whole log files.10:25
*** bootsha has joined #openstack-kolla10:26
*** Mr_Broken has joined #openstack-kolla10:26
*** neilus has joined #openstack-kolla10:28
sean-k-mooneytyrola: did you create the router manually or with the init_runonce script?10:28
tyrolamanually, tried via gui and cli10:29
sean-k-mooneyon your network node can you see the network namespace of the router?10:30
tyrolahttp://paste.openstack.org/show/AzG6QJJDnZgmuW02k7bq/ (commands I've used)10:30
*** tfukushima has quit IRC10:32
tyrolaYap, just on the 2nd node. One node has only one for dhcp and one has 2 (dhcp and router)10:32
sean-k-mooneyok the command look correct but i belive the uplink port on the will not have the gateway ip so you will have assign 5.83.160.1 to the br-ex on the node that has qouter namespace10:33
tyrolaand the router namespace has 3 interfaces, lo, one for internal ips, and one floating ip interface10:34
tyrolaThe subnet has already a external router from our datacenter with ip 5.83.160.110:34
tyrolathis router is maintained by our carrier10:35
sean-k-mooneyok yes that should work too. am from the router network namespace can you ping that upstream router ip?10:35
sean-k-mooneysudo ip netns exec <route ns> ping <upstream router>10:36
tyrolaOkay, not that isn't working.. maybe the bridges aren't correct?10:36
tyrolaIn which state must be each bridge on a working setup? (br-int, br-tun and br-ex)?10:37
sean-k-mooneyi assume you added an interface directly to the br-ex bridge instead of nating it with iptables10:38
sean-k-mooneythe local ports on the br-* interfaces do not need to be up10:38
*** prithiv has joined #openstack-kolla10:38
sean-k-mooneyyou would only bringe them up for debugging10:38
tyrolaDoes kolla add the external interface normally to br-ex?10:39
sean-k-mooneythe interfaces that are directly added to the br-ex do need to be brought up in the kernel10:39
sean-k-mooneyno10:39
tyrolaOkay.. that should be already the reason. damn10:39
sean-k-mooneywell10:39
sean-k-mooneykolla will a an interface that will be used by the vm data network to the br-ex10:40
sean-k-mooneyis should have been more clear10:40
sean-k-mooneythat interface can be used for upstream internet traffic also10:40
sean-k-mooneykolla will not change the interface state however10:40
sean-k-mooneyit will jsut add it to the ovs bridge10:41
sean-k-mooneycan you ifcofnig on the interface that is added to the br-ex and check that it is up10:41
tyrolaI will test it and give you feedback in a few minutes :) Thanks10:42
*** daneyon has joined #openstack-kolla10:42
*** daneyon has quit IRC10:46
*** gfidente has joined #openstack-kolla10:47
*** fragatina has quit IRC10:48
*** fragatina has joined #openstack-kolla10:49
*** mewald has quit IRC10:49
tyrolaIt seems like the br-XX interfaces on the neutron nodes aren't bridges10:53
tyrolabrctl show doesn't shows them up correctly. what the best and correct way to configure the network? Can't find anything in the kolla documentation for the bridge setup.10:54
tyrolaBut if I run inside the openvswitch container "ovs-vsctl show" it shows up the br-ex interface with Port external10:56
sean-k-mooneythe bridge should not be shown in brctl you should use "ovs-vsctl show" to view the ovs bridges10:56
sean-k-mooneyyes10:56
sean-k-mooneyyou will need to be in the ovs container to see the bridge config10:56
sean-k-mooneywhen you look at the br-ex you should see the interface specified in neutron_external_interface: "eth4" added to the bridge10:57
*** sacharya has joined #openstack-kolla10:58
tyrolahttp://paste.openstack.org/show/hX3yiMl48kiF9EC7u3Y7/ full output10:58
*** salv-orl_ has joined #openstack-kolla10:59
sean-k-mooneyi take it you set  neutron_external_interface: external in the global.yml11:00
tyrolayap thats correct, the full interfaces list http://paste.openstack.org/show/a9BTlh0xt4HHnAZGkKeE/11:00
*** salv-orl_ has quit IRC11:01
*** salv-orlando has quit IRC11:02
*** sacharya has quit IRC11:02
sean-k-mooneyhum the interface appears to be up. so looking at the vsctl out put you should have a flow to strip vlan tag 4 and output normal on br-ex11:04
sean-k-mooneysudo ovs-ofctl dump-flow br-ex11:05
sean-k-mooneyyou should also have a reverse flow that will add vlan4 if traffic coming in port int-br-ex untagged on the br-int11:06
sean-k-mooneyare they present?11:06
*** Mr_Broken has quit IRC11:07
*** tyrola has quit IRC11:08
*** tyrola has joined #openstack-kolla11:09
tyrolahttp://paste.openstack.org/show/242g0hxd3lOBHcjqVoZf/11:09
tyrolaWhy vlan tag 4? where is this set?11:11
sean-k-mooneyPort "qg-2d69639a-b5"11:12
sean-k-mooney            tag: 411:12
sean-k-mooney            Interface "qg-2d69639a-b5"11:12
sean-k-mooneyqg => quantum(old name for neutron) gateway11:12
tyrolaMust the tag id match with the external network vlan tag id? or is this separated with the namespace?11:13
sean-k-mooneythe tag id is purly internal for tenent isolation11:13
tyrolaOk alright11:13
sean-k-mooneyneutron will translate it to the appropriate external vlan11:13
sean-k-mooneyin this case you created a flat network11:13
sean-k-mooneyso no vlan is used11:13
*** Mr_Broken has joined #openstack-kolla11:14
*** g3ek has quit IRC11:14
*** haplo37 has quit IRC11:14
sean-k-mooneylooking at the flow on br-ex they are correctly set up11:14
sean-k-mooneycookie=0x92695e18ab45a9ea, duration=2748.342s, table=0, n_packets=73, n_bytes=3066, idle_age=288, priority=4,in_port=2,dl_vlan=4 actions=strip_vlan,NORMAL11:15
sean-k-mooneythis flow is being hit11:15
sean-k-mooneyso traffic should be getting to your phyical network untagged11:15
*** haplo37 has joined #openstack-kolla11:16
*** g3ek has joined #openstack-kolla11:16
sean-k-mooneyif you ran tcpdump -i external and ping the datacenter router form the neutron router namespace you should see the arp request for the upstream router mac11:17
*** zhiwei has joined #openstack-kolla11:17
sean-k-mooneydoes your tor require the traffic to be vlan tagged with a specific vlan to reach the upstream rourter maybe?11:17
tyrolaMh ok. I need to say I am completely new to network namespaces and neutron itself.11:18
*** Jeffrey4l_ has quit IRC11:18
tyrolaNo vlan tags needed, the vlan's are managed by the router.11:19
sean-k-mooneyam just as a quick test to double check can you try pinging the upstream router and confirm the packets are leaveing the host with tcpdump?11:20
tyrolahttp://paste.openstack.org/show/foxBRwJe2fIAYpMwYCML/ arp request seems to work11:20
sean-k-mooneycool11:20
tyrolatcpdump was running outside of the docker container11:20
sean-k-mooneyso it looks like you are not getting an arp reply11:20
*** zhiwei has quit IRC11:21
sean-k-mooneyam out of interest were you able to reach the router successfully before you deployed kolla?11:21
tyrolaI am able to reach the router everywhere just not inside of the router namespace11:22
tyrolahttp://paste.openstack.org/show/lAqCJhIIr3I4di2eEtMz/ as you can see it just doesn't work if I am inside the qrouter namespace11:22
sean-k-mooneyout side of the router namespace you would be using a different network interface though11:23
sean-k-mooneyi would guess your are reaching it via the internal interface11:24
tyrola[root@neutron02 ~]# ping 5.83.160.1 -I external11:24
tyrolaping: Warning: source address might be selected on device other than external.11:24
tyrolaPING 5.83.160.1 (5.83.160.1) from 10.0.0.22 external: 56(84) bytes of data.11:24
tyrolaFrom 10.0.0.22 icmp_seq=1 Destination Host Unreachable11:24
tyrolaFrom 10.0.0.22 icmp_seq=2 Destination Host Unreachable11:24
tyrolaIt seems you are right :(11:24
sean-k-mooneyim assuming this is a production setup. if not the simple answer is to nat the traffic11:25
tyrolaYes, this should be a production env soon11:26
openstackgerritPaul Bourke proposed openstack/kolla: Document a common Ceph bootstrap failure scenario  https://review.openstack.org/33442211:29
*** mewald has joined #openstack-kolla11:30
*** Jeffrey4l_ has joined #openstack-kolla11:30
tyrolaYes, this should be a production env soon11:32
*** Mr_Broken has quit IRC11:36
*** Mr_Broken has joined #openstack-kolla11:37
*** mliima has joined #openstack-kolla11:38
*** Jeffrey4l_ has quit IRC11:38
*** neilus1 has joined #openstack-kolla11:39
tyrolasean-k-mooney: Seems like the switch ports aren't in the correct vlan... we will fix it asap I think thats the issue11:39
tyrolaThank you very much for helping me. It not fixed our issue but I've learned how to debug neutron a little bit more :)11:39
*** neilus has quit IRC11:40
*** neilus1 has quit IRC11:47
openstackgerritMerged openstack/kolla: Add unit test for stop and restart container  https://review.openstack.org/33152411:51
*** Mr_Broken has quit IRC11:51
*** Mr_Broken has joined #openstack-kolla11:54
sean-k-mooneytyrola: glad i could help. hopefully you will have it working soon11:55
*** haplo37 has quit IRC11:55
mandrepbourke: not sure my -1 for workflow is going to change anything in https://review.openstack.org/#/c/334237/11:57
patchbotmandre: patch 334237 - kolla (stable/liberty) - Create ansible home directory in kolla-toolbox con...11:57
*** Jeffrey4l_ has joined #openstack-kolla11:58
pbourkeshould do11:58
pbourkeremoved mine too11:58
*** rhallisey has joined #openstack-kolla12:00
*** neilus has joined #openstack-kolla12:04
*** haplo37 has joined #openstack-kolla12:04
*** diga has quit IRC12:11
*** williamcaban has joined #openstack-kolla12:14
bootshahi Martin and Paul, thanks for the review of https://review.openstack.org/#/c/334237/. Sorry for my ignorance, but As suggested by Martin, Should I remove the exiting patch set completely and submit a fresh one with git cherry-pick -x ?12:15
patchbotbootsha: patch 334237 - kolla (stable/liberty) - Create ansible home directory in kolla-toolbox con...12:15
*** ppowell_ has joined #openstack-kolla12:23
openstackgerritMerged openstack/kolla: Document a common Ceph bootstrap failure scenario  https://review.openstack.org/33442212:28
rhalliseybootsha, sure that works12:28
mandrebootsha: you don't have to abandon your current patch if you modify the commit message to add a line containing (cherry-picked from commit ...)12:29
*** daneyon has joined #openstack-kolla12:30
bootshagot it thanks. I will probably add cherry-picked from commit ...12:30
mandrebootsha: in your case, you need to add (cherry-pick from commit 76f97b406735540eb86fb3f293343ed4803c45d7)12:30
mandrebootsha: fyi, that's the only thing that 'git cherry-pick -x' does for you, it adds this line to the commit message12:31
bootshayea, I saw that, but was in doubt :)12:31
*** daneyon has quit IRC12:34
openstackgerritMohammed Salih Puthenpurayil proposed openstack/kolla: Create ansible home directory in kolla-toolbox container  https://review.openstack.org/33423712:39
openstackgerritMohammed Salih Puthenpurayil proposed openstack/kolla: Create ansible home directory in kolla-toolbox container  https://review.openstack.org/33423912:40
*** bootsha has quit IRC12:41
*** zhiwei has joined #openstack-kolla12:42
*** haplo37 has quit IRC12:44
*** salv-orlando has joined #openstack-kolla12:44
*** ccesario has joined #openstack-kolla12:45
*** g3ek has quit IRC12:45
*** haplo37 has joined #openstack-kolla12:46
*** g3ek has joined #openstack-kolla12:46
*** zhiwei has quit IRC12:47
*** salv-orlando has quit IRC12:51
*** matrohon has joined #openstack-kolla12:52
*** williamcaban has quit IRC12:55
tyrolasean-k-mooney: works now fine :) thanks again12:55
*** williamcaban has joined #openstack-kolla12:57
sean-k-mooneyno worries. i hit a similar issue before glad i could help13:00
*** coolsvap has quit IRC13:00
*** klint has quit IRC13:01
*** williamcaban has quit IRC13:01
*** Mr_Broke_ has joined #openstack-kolla13:07
*** Mr_Brok__ has joined #openstack-kolla13:09
*** sdake has joined #openstack-kolla13:10
*** Mr_Broken has quit IRC13:10
*** Mr_Broke_ has quit IRC13:11
*** jtriley has joined #openstack-kolla13:14
*** williamcaban has joined #openstack-kolla13:16
*** zhurong has joined #openstack-kolla13:18
*** sdake_ has joined #openstack-kolla13:20
*** sdake has quit IRC13:21
*** Mr_Brok__ has quit IRC13:27
*** Mr_Broken has joined #openstack-kolla13:27
*** inc0 has joined #openstack-kolla13:28
*** diogogmt has quit IRC13:29
*** haplo37 has quit IRC13:29
*** g3ek has quit IRC13:30
*** belmoreira has quit IRC13:31
*** Mr_Broke_ has joined #openstack-kolla13:34
*** haplo37 has joined #openstack-kolla13:35
*** g3ek has joined #openstack-kolla13:35
*** Mr_Broken has quit IRC13:36
*** Mr_Broken has joined #openstack-kolla13:38
*** Mr_Brok__ has joined #openstack-kolla13:40
*** Mr_Broke_ has quit IRC13:40
*** banix has joined #openstack-kolla13:41
*** Mr_Broken has quit IRC13:42
*** inc0 has quit IRC13:44
*** inc0 has joined #openstack-kolla13:44
*** aernhart has joined #openstack-kolla13:45
*** kangh_ has joined #openstack-kolla13:46
sdake_morning13:47
kangh_hey sdake13:47
*** salv-orlando has joined #openstack-kolla13:48
inc0howdy folks13:49
mliimamorning all13:49
*** Mr_Broken has joined #openstack-kolla13:52
*** b_bezak has quit IRC13:52
*** Mr_Brok__ has quit IRC13:55
*** prithiv has quit IRC13:55
*** Mr_Broke_ has joined #openstack-kolla13:55
*** kproskurin has joined #openstack-kolla13:55
*** salv-orlando has quit IRC13:55
*** kangh_ has left #openstack-kolla13:56
*** bootsha has joined #openstack-kolla13:56
*** Mr_Broken has quit IRC13:57
openstackgerritMohammed Salih Puthenpurayil proposed openstack/kolla: Trivial fixes to correct couple of typos.  https://review.openstack.org/33430613:58
*** huikang has joined #openstack-kolla13:59
*** Mr_Broke_ has quit IRC14:04
*** Mr_Broken has joined #openstack-kolla14:04
*** wmiller has quit IRC14:05
*** wmiller has joined #openstack-kolla14:05
*** Mr_Broke_ has joined #openstack-kolla14:09
*** Mr_Broken has quit IRC14:11
*** diogogmt has joined #openstack-kolla14:12
*** bootsha has quit IRC14:13
*** zhurong has quit IRC14:13
*** b_bezak has joined #openstack-kolla14:15
*** haplo37 has quit IRC14:16
*** g3ek has quit IRC14:17
*** g3ek has joined #openstack-kolla14:17
*** daneyon has joined #openstack-kolla14:18
*** haplo37 has joined #openstack-kolla14:18
*** Mr_Broken has joined #openstack-kolla14:20
*** ayoung has joined #openstack-kolla14:22
*** Mr_Brok__ has joined #openstack-kolla14:22
*** Mr_Broke_ has quit IRC14:22
*** daneyon has quit IRC14:23
*** Mr_Broken has quit IRC14:25
*** ssurana has joined #openstack-kolla14:27
*** salv-orlando has joined #openstack-kolla14:30
*** Mr_Brok__ has quit IRC14:36
*** Mr_Broken has joined #openstack-kolla14:43
*** zhiwei has joined #openstack-kolla14:44
*** ayoung_ has joined #openstack-kolla14:48
*** zhiwei has quit IRC14:48
*** JeroenBo has quit IRC14:49
*** belmoreira has joined #openstack-kolla14:50
*** david-lyle_ has joined #openstack-kolla14:51
openstackgerritSteven Dake proposed openstack/kolla: Test of mitaka gate  https://review.openstack.org/33451914:55
*** david-lyle has quit IRC14:55
sdake_does anyone have any idea why this gat efialure is occuring14:56
sdake_http://logs.openstack.org/10/333110/1/check/gate-kolla-dsvm-deploy-centos-binary/a2673af/console.html.gz#_2016-06-23_05_32_19_56736214:56
sdake_it  happens all the time14:57
*** sacharya has joined #openstack-kolla14:57
*** sdake has joined #openstack-kolla15:01
*** mliima has quit IRC15:01
*** mewald has quit IRC15:01
*** mewald has joined #openstack-kolla15:02
*** aernhart has quit IRC15:02
sdakeinc0 extradb15:03
sdakei want to talk about it15:03
sdakextradb15:03
sdakeor whatever it is15:03
inc0you mean percona stuff?15:03
sdakeright15:03
sdakedo we need it?15:03
*** sdake_ has quit IRC15:03
*** aernhart has joined #openstack-kolla15:03
inc0well, it does provide some cool features15:03
sdakeanother way to word that is15:04
inc0I don't have much experience with it15:04
sdakecan we operate without it15:04
inc0we operate without it now15:04
sdakextradb is definately used15:04
inc0what percona gives you is for example simple backups15:04
sdakecool so that wfm but i want to operate without it15:04
inc0ok, it is there15:07
sdakei dont even know what it does15:07
inc0it's a toolset for mariadb15:07
inc0that helps you with certain tasks15:07
sdakewhen i was debugging it about 6 months ago because it was busted15:07
inc0not required at all afaik15:07
sdakei noticed only one part of percona is used15:09
inc0xtrabackup right15:09
*** mewald has quit IRC15:09
inc0?15:09
sdakeyup15:09
sdakextradbbackup i think15:09
inc0yeah, this helps with bit-level backup15:09
sdakekolla doesn't work at all without it15:09
inc0not sure if galera uses this15:09
inc0it might actually15:09
sdakei know other peopel have gotten replication with galera to work without it15:09
inc0soo...I'd lie if I'd say that this is my area of expertise15:09
*** david-lyle_ is now known as david-lyle15:09
sdakedid you add it?15:09
inc0no. Sam did15:09
sdakei see15:09
inc0so I don't think it needs15:09
sdaketrying to come up for air here, any chance anyone can look into removing it15:09
inc0percona15:09
sdakeor making it optional?15:09
*** prithiv has joined #openstack-kolla15:09
inc0galera doesn't seem to need percona15:09
sdakekolla crashes without xtradb15:09
inc0but how crashes? what's the error?15:09
sdakei dont recall it was 6 months ago i looked at it15:09
sean-k-mooneysdake: it should be able to run with InnoDB15:09
sdakesean-k-mooney morning fine sir15:09
sdakesean-k-mooney so my diagram was a talking point15:09
sean-k-mooneyxtradb is a perfromace optimsed version of InnoDB15:09
*** ayoung has quit IRC15:09
inc0https://github.com/openstack/kolla/blob/6e8f01ca6a02f77a281df78f544cace063c78511/ansible/roles/mariadb/templates/wsrep-notify.sh.j2#L60 sdake15:09
sean-k-mooneysdake: and yes morning15:09
*** ayoung_ is now known as ayoung15:09
sdakesean-k-mooney if you think it should be done a different way please let me knwo so i can modify the digram15:09
inc0I think if you change config of galera to not use xtrabackup, it might work15:10
*** mliima has joined #openstack-kolla15:10
*** belmoreira has quit IRC15:10
*** tyrola has quit IRC15:11
sdakeany idea how to do that15:11
sdakeI dont know jack shit about galera15:11
*** tyrola has joined #openstack-kolla15:11
wirehead_kfox1111_: yeah, I know we need to address the issue of PersistentVolumeClaims but haven't worried about it yet.15:12
sdakewtb new redidck movie15:12
sdakemorning wirehead_15:12
wirehead_kfox1111_: If we were to be starting over from zero without the existing working Kolla codebase, I'd totally try to build Kolla-kubernetes around Helm instead of Ansible.15:13
wirehead_gmorning15:16
sean-k-mooneysdake: i have started working on rebasing the bifrost work today.15:16
*** fragatina has quit IRC15:17
*** fragatina has joined #openstack-kolla15:17
sean-k-mooneysdake: just googleing around i found this https://mariadb.com/kb/en/mariadb/mariadb-galera-cluster-known-limitations/15:18
sean-k-mooneysdake: first bullet point "Currently replication works only with the InnoDB storage engine. "15:18
sean-k-mooneyso you will need to have innoDB or xtraDB to use galara15:18
*** ssurana has quit IRC15:21
*** Mr_Broken has quit IRC15:22
*** Mr_Broken has joined #openstack-kolla15:22
openstackgerritMerged openstack/kolla: Trivial fixes to correct couple of typos.  https://review.openstack.org/33430615:22
*** coolsvap has joined #openstack-kolla15:24
Jeffrey4l_sdake, morning15:27
openstackgerritMerged openstack/kolla: Add the verification of required images step before the deployment.  https://review.openstack.org/33366615:27
Jeffrey4l_I create a test patch to build the image with ansible. https://review.openstack.org/334208 could u give us some idea for this?15:27
coolsvapJeffrey4l_, hi there is no such rule as such15:28
coolsvapbut yes i would like to have more eyes on it for sure15:29
Jeffrey4l_coolsvap, thanks.15:29
*** matrohon has quit IRC15:29
Jeffrey4l_the root cause i want to use ansible is: we need defend program in the dockerfile.15:30
*** dmk0202 has quit IRC15:30
Jeffrey4l_Recently, the ceph packaging changed and it create ceph user now, which crashed the kolla.15:30
Jeffrey4l_if using ansible, the solution is easy, just: `user: name=ceph`.15:31
*** harlowja_at_home has joined #openstack-kolla15:31
*** daneyon has joined #openstack-kolla15:31
Jeffrey4l_if using raw dockerfile/RUN, we need `RUN id -u ceph &>/dev/null || useradd --user-group ceph`.15:31
Jeffrey4l_which is ugly.15:31
coolsvapJeffrey4l_, personally i like the approach it is very much in the lines for the elemental discussion we had15:32
Jeffrey4l_the second reason is, ansible is more powerful.15:32
ccesariohey guys... due the needed of a cinder SVC driver, I need install multipathd into  compute-node container (using the inc0 customize path) .... but I need that this service starts automatic, does someone know what is the best way of to do it ?15:32
Jeffrey4l_yes. it will resolve the customize issue when building image.15:33
*** daneyon_ has quit IRC15:34
*** Serlex has quit IRC15:35
*** ssurana has joined #openstack-kolla15:37
*** matrohon has joined #openstack-kolla15:38
openstackgerritDipa Thakkar proposed openstack/kolla: Remove MariaDB warning from HAproxy  https://review.openstack.org/33405415:38
*** Mr_Broken has quit IRC15:39
sdakeJeffrey4l_ yo15:40
Jeffrey4l_sdake, could you check above words i said?15:41
sdakeJeffrey4l_ i just read scrollback15:41
sean-k-mooneyJeffrey4l_:  why do that check instead of using the ansible user module15:41
sdakei have your review up15:41
sdakelooks daunting :)15:41
Jeffrey4l_thanks.15:42
sean-k-mooneyJeffrey4l_: never mind missread  RUN id -u ceph &>/dev/null || useradd --user-group ceph is in the docker file not ansible15:42
*** pbourke has quit IRC15:43
*** pbourke has joined #openstack-kolla15:43
Jeffrey4l_sean-k-mooney, :)15:43
*** jmccarthy has joined #openstack-kolla15:44
sean-k-mooneyccesario if you want a standalone server to run on the compute node you should create a container and add it to the compute group15:44
sean-k-mooneys/server/service/15:45
*** haplo37 has quit IRC15:45
*** ssurana has quit IRC15:51
*** g3ek has quit IRC15:52
*** prithiv has quit IRC15:52
*** g3ek has joined #openstack-kolla15:54
sdakeit sure would be handy if pbr was someway used to generate tag ids for stable branches15:54
sdakeJeffrey4l_ had a look at the build work you did or ansible15:54
sdakei'm curious, what is the purposoe15:54
sdakeorfor15:54
sdakeis this a third alternative to the dsl and the customization via docker.j2?15:54
*** Mr_Broken has joined #openstack-kolla15:54
Jeffrey4l_1. Defensive programming: Recently, the ceph packaging changed and it create ceph user now, which crashed the kolla.15:54
Jeffrey4l_if using ansible, the solution is easy, just: `user: name=ceph`.15:54
Jeffrey4l_if using raw dockerfile/RUN, we need `RUN id -u ceph &>/dev/null || useradd --user-group ceph`.15:54
Jeffrey4l_there are two much such potential issue exist.15:55
sdakeso your proposing rewriting all 130 dockerfiles as ansible?15:55
*** haplo37 has joined #openstack-kolla15:55
Jeffrey4l_2. it will resolve the customize issue when building image. and yes it just like a kind of DSL language.15:56
*** b_bezak has quit IRC15:56
sdakewhat about build performance15:56
Jeffrey4l_yes. if the base is acceptable. I think we'd better change all the dockfiles.15:56
sdakewe have a bunch of cli options to build.py15:57
Jeffrey4l_sdake, the performance should almost the same. ansible do not add much logical.15:57
sdakehow to implement those?15:57
sdakeJeffrey4l_ our build is multitheaded15:57
sdakeansible not right?15:57
Jeffrey4l_sdake, the ansible is run in one container. so we can build multi container in one time.15:57
sdakemy biggest concern is loss of all of the CLI options to build.py15:58
sdakeinc0 ^^15:58
Jeffrey4l_most of the option is use by the build.py script. there is few is used by the dockerfile.15:59
Jeffrey4l_see this https://github.com/openstack/kolla/blob/master/kolla/image/build.py#L626,L63815:59
Jeffrey4l_now, there only 14 value is used in the dockerfile.15:59
Jeffrey4l_and we can pass the to the ansible-play by using `-e aaa=bbb` format.16:00
Jeffrey4l_ansible-playbook16:00
* inc0 reading log16:00
inc0Jeffrey4l_, you know our customization mechanism right?16:00
Jeffrey4l_inc0, yes.16:01
inc0so when ceph updated this user thign16:01
inc0we could add this into our Dockerfiles16:01
inc0and that's it right?16:01
Jeffrey4l_normally, kolla should handle this.16:02
Jeffrey4l_rather than fix by us.16:02
inc0Jeffrey4l_, please repeat the issue, I don't really get it16:02
Jeffrey4l_Defensive programming is important16:02
Jeffrey4l_inc0, https://review.openstack.org/33416816:02
Jeffrey4l_here is the fix.16:02
inc0what does defensive programming mean?16:03
Jeffrey4l_inc0, https://en.wikipedia.org/wiki/Defensive_programming16:03
Jeffrey4l_inc0, ceph issue http://logs.openstack.org/58/334158/1/check/gate-kolla-dsvm-build-ubuntu-source/c28e596/console.html#_2016-06-25_10_18_07_94981516:05
*** mewald has joined #openstack-kolla16:05
inc0ok I see16:06
*** daneyon_ has joined #openstack-kolla16:06
inc0how is it possible that ceph user exists on newly built container?16:07
*** Mr_Broken has quit IRC16:07
Jeffrey4l_i think someone/canical changed the ceph packages. and when installing ceph, the package will create the ceph user automatically.16:08
*** Mr_Broken has joined #openstack-kolla16:08
*** vhosakot has joined #openstack-kolla16:09
*** sacharya has quit IRC16:09
Jeffrey4l_there is another approach.16:09
Jeffrey4l_just like:16:09
Jeffrey4l_1. run a container with `sleep infinity`16:09
Jeffrey4l_2. using ansible + docker connection to  initial the containers. ( install package and prepare the container env, just like it is a LXC)16:10
sdakehttp://paste.fedoraproject.org/385565/46704382/ -> http://paste.fedoraproject.org/385565/4670438216:10
sdakeliberty is broked^16:10
Jeffrey4l_3. commit the container to a tag.16:10
sean-k-mooneyJeffrey4l_: what benifit does that have over a docker build?16:11
*** daneyon_ has quit IRC16:11
Jeffrey4l_in this way, there is not Dockerfile anymore. there should be only one ansible playbooks, which will generate the kolla images.16:11
Jeffrey4l_And we can reuse/extend the playbooks in different image.16:12
inc0Jeffrey4l_, I'm -2 to this, we need containers to be ansible-agnosticv16:12
*** sacharya has joined #openstack-kolla16:12
sdakewell you can't really be -2 to a spec16:12
sdakebut i dont see a strong value or this work16:12
inc0we already have services consuming just containers16:12
sean-k-mooneysdake: the only value i see with that approch would be if we wanted to support a different runtime e.g. rocket or appc16:13
Jeffrey4l_sean-k-mooney, scrollback..I told serveral benefit.16:13
sdakeJeffrey4l_ what i'm getting at is the epain doesnt seem worth the gain16:13
sdakewe need customizations16:14
Jeffrey4l_sdake, i will push a follow patch to describle how the customization will looks like .16:14
sdakei am not super keen to rework the entire dockerfile syntax into something completely different that isn't coherent for what peopel are trained for 3 months before release:)16:14
*** jmccarthy has left #openstack-kolla16:14
sdakecn someon look at that keystone log i pasted pls16:15
sdaketip of stable/liberty16:15
sdakejust built 15 minutes ago or so16:15
Jeffrey4l_sdake, seem the keystone db is not synced.16:15
*** ayoung has quit IRC16:16
sdakeya it looked like bootstrap didnt go to me too16:16
sdaketrying a redeploy16:16
Jeffrey4l_sdake, need remove the keystone database. Or the whole db16:16
sdakei did a total cleanup16:17
sdakethis is a fresh bulid16:17
sdakeTASK: [keystone | Running Keystone bootstrap container] ***********************16:18
sdakeskipping: [localhost]16:18
sdakeTASK: [keystone | Starting keystone container] ********************************16:18
sdakechanged: [localhost]16:18
sdakeTASK: [keystone | Wait for keystone startup] **********************************16:18
sdakeok: [localhost]16:18
sdakesorry for spam16:18
sdakewhy would it skip bootstrap?16:18
Jeffrey4l_skipping..16:18
Jeffrey4l_copy the above log out.16:20
Jeffrey4l_sdake, ^16:20
sdakehuh?16:20
sdakeyou mean paste entire output?16:20
Jeffrey4l_right. the logs before Running Keystone bootstrap container16:21
Jeffrey4l_just 10-20 log should be enough.16:21
sdakeansible 1.9.4 on tip of stable/liberty16:22
sdakehttp://paste.fedoraproject.org/385574/4454514/ -> http://paste.fedoraproject.org/385574/0445451416:22
*** Mr_Broken has quit IRC16:22
sdakeseems to be skipping lots of things it should not16:23
Jeffrey4l_sdake, i do not think you db is cleaned.16:24
Jeffrey4l_TASK: [keystone | Creating Keystone database] *********************************16:24
Jeffrey4l_[0;32mok: [localhost -> localhost][0m16:24
sdakecleanup-containers - cleanup-host - cleanup-images16:24
Jeffrey4l_this taks is OK rather than change. means you keystone database exist.16:24
*** Mr_Broken has joined #openstack-kolla16:25
*** coolsvap has quit IRC16:26
sdakethere is no database on my system16:27
inc0sdake, are volumes deleted?16:28
sdakelet me check moment16:29
*** coolsvap has joined #openstack-kolla16:29
inc0maybe you have stale volumes and that makes keystone bootstrap skip16:29
sdakehttp://paste.fedoraproject.org/385578/46704497/ -> http://paste.fedoraproject.org/385578/4670449716:29
sean-k-mooneyinc0: the cleanup contaiers script is ment to delete the volumns too right16:29
inc0yeah, but sometimes fail to do so16:30
inc0we had some fixes in master16:30
*** ayoung has joined #openstack-kolla16:31
*** g3ek has quit IRC16:34
*** haplo37 has quit IRC16:34
sdakewell i just wiped out /vaar/lib/docker16:35
sdakelets see if it workie now16:35
sean-k-mooneysdake: thats rather a blunt hammer16:36
sdakeI'LL DROP THE HAMMER16:36
*** coolsvap has quit IRC16:37
*** haplo37 has joined #openstack-kolla16:37
*** harlowja_at_home has quit IRC16:37
*** g3ek has joined #openstack-kolla16:37
*** tyrola has quit IRC16:37
openstackgerritMd Nadeem proposed openstack/kolla: Add test for remove container, get state and get env  https://review.openstack.org/33155616:38
sdakeJeffrey4l_ ar you tlakingbout merging our deploy and bulid steps?16:41
*** athomas has quit IRC16:46
openstackgerritMerged openstack/kolla: Create ansible home directory in kolla-toolbox container  https://review.openstack.org/33423916:48
sdakethis weekend i hit 230 in the world for diablo barbarian16:48
*** matrohon has quit IRC16:49
sdakecracking the top 100 is near impossible - those cats must play 12 hours a daay16:50
sdakei beat the timer by 0.117 seconds16:50
*** aNupoisc has joined #openstack-kolla16:51
sdakebootstrap skipped again - completely clean /var/lib/docker16:52
*** ntpttr_ has quit IRC16:56
sdakewould someone else try the tip of liberty?16:59
*** salv-orl_ has joined #openstack-kolla16:59
*** belmoreira has joined #openstack-kolla17:00
jogammorning gents, sorry to interrupt, but are the latest kolla/ubuntu-source-* images on docker hub compiled on/for Ubuntu 14.04 LTS17:00
sdakejogan we are in the process of moving to the kolla namespace17:01
sdakejogam so i am not sure wha tthe status is there - coolsvap has been doing the workk17:01
jogamADUM: are there any images (latest being 3.0.0) for 16.04 LTS or would you avoid ubuntu17:01
sdakei would avoid master for now17:02
sdakeand use mitaka (2.0.1)17:02
jogamsdake: was looking at the kolla/* namespace on docker, was that the one you meant with the move destination?17:02
*** salv-orlando has quit IRC17:02
sdakejogam yup17:02
*** belmoreira has quit IRC17:03
sean-k-mooneysdake: dose mitaka currently deploy with the stable mitaka branch of ansible? i had to swap to master to get it to work the last time i tried to deploy mitaka images17:04
sdakeit should17:04
sdakewhether it does or not is unknown at this time17:04
sdakei am trying liberty atm and it seems broken17:04
*** ayoung has quit IRC17:06
openstackgerritMerged openstack/kolla: Enable openvswitch container logs in host volumes  https://review.openstack.org/33428317:06
jogamsdake or dockerhub pushers: what would you recommend to use for an eval system setup with three nodes to convince management to use kolla for OS deploy? (2.0.1 from DH/kolla, compile from current master, 3.0.0 from DH/kolla)?17:10
sdakejogan do following17:11
sdakegit clone http://github.com/openstack/kolla17:11
*** zhiwei has joined #openstack-kolla17:11
sdakegit checkout 2.0.117:11
sdaketools/build.py --registery YOUR_REGISSTRY_SERVER:4000 --push17:11
jogamawesome, e.g. build 2.0.1 from latest git clone17:11
sdakein other owrds, compile 2.0.1 and push to a local registry17:11
jogamscrolling throught he list this morning I saw some issues with CEPH, after having failed with Cinder on 16.04 LTS, does CEPH work on the 2.0.1 branch on virtual env with dual virt. HD?17:13
jogamand 14.04 LTS?17:13
sdakeit worked when we released it17:13
sdakedont now about 16.0417:13
sdakewe dont support 16.04 in stable branches17:13
jogamI learned that the hard way :(17:13
sdakeit "should" work but if it does or not I dont know17:14
jogamBUT: the highest ´likelyness´ that it works is currently with 2.0.1 from git latest?17:14
sdake2.0.1 is a tag17:15
sdakeso at that point atleat 6 people manually testedd it17:15
sdakeand it worked17:15
sdakebut the packaging changes17:15
sdakeso that breaks tags unfortunately17:15
*** zhiwei has quit IRC17:15
sdakejogam you don't want to deploy maste rbecause that deploys ALL OF OPENSTACK as master17:17
sdakejogam your use case (eval) is best served by a stable branch17:18
sdakejogam which deploys either liberty (1.1.1) or mitaka (2.0.1)17:18
jogamok, slowly starting to get your GitHub setup, somewhat :)17:19
*** ssurana has joined #openstack-kolla17:20
sdakewhere is the ceph resolution that Jeffrey4l_ was talkingabout?17:21
sdakejogan 3.0.0 is not yet released17:21
sdakewe dont push to the docker hub any unreleased software17:21
*** ayoung has joined #openstack-kolla17:22
*** Mr_Broken has quit IRC17:22
jogamsdake: already there --> https://hub.docker.com/r/kolla/ubuntu-source-heka/tags17:23
*** Mr_Broken has joined #openstack-kolla17:23
sdakejogam it is going to be deleted17:23
sdakecoolsvap did wrong when he pushed the 3.0.0 images17:23
*** g3ek has quit IRC17:23
*** jtriley has quit IRC17:23
*** haplo37 has quit IRC17:23
jogamok, no worries, I´ll shut up and just monkey around here and compile the latest master and mitaka branches and then see what happens17:24
*** g3ek has joined #openstack-kolla17:25
*** harlowja has quit IRC17:25
sdakefeel free to ask questions :)17:25
sdakei wouldn't use master though17:25
*** haplo37 has joined #openstack-kolla17:25
jogamsdake: one more, for mitaka its still ansible < 2.0.0?17:25
sdakebecause openstack master is always in a state of brokeness17:25
sdakeright mitaka and liberty are ansible 1.9.417:26
jogamoh sweet, so everyone who is using OS is keeping bravely a couple of releases back?17:26
*** harlowja has joined #openstack-kolla17:33
*** Mr_Broken has quit IRC17:40
openstackgerritSteven Dake proposed openstack/kolla: Use proper images when deploying mitaka  https://review.openstack.org/33458817:43
sdakejogam i just confirmed liberty is working17:45
sdakegit checkout stable/liberty17:45
*** salv-orl_ has quit IRC17:47
*** fragatina has quit IRC17:49
*** Mr_Broken has joined #openstack-kolla17:53
*** daneyon_ has joined #openstack-kolla17:55
jogamsdake: thx, will go for that then with ansible 1.9.4, anything else version-wise that I should NOT do :)17:55
sdakedocker 1.10 or later17:55
sdakejogam also change the #openstack_version: whateveritis" in /etc/kolla/globals.yml to openstack_version: "1.1.2"17:56
sdakeif you use git checkout 2.0.117:56
sdakeit will use the correcct image and you wont need to do this step17:57
jogamhuh? so when git checkout 2.0.1 tag I get the liberty release which is 1.1.2??17:57
sdakenoep you get a release that is 1.1.117:57
sdakebut after that, pbr autoincrementst he versoin to 1.1.217:57
sdakestable/liberty != 1.1.117:58
sdakestable/liberty has critical and high bug fixes17:58
sdakewe release z streams every 45 days17:58
*** mewald has left #openstack-kolla17:58
jogamsdake: so you meant stable/mitaka is working above?17:59
*** daneyon_ has quit IRC17:59
sdakestable/mitaka != 2.0.117:59
sdakestable/mitaka gets the same bugfixes as stable/liberty17:59
sdakethey ar essnetially teh same codebase18:00
jogamOK, so what branch is the tag 2.0.1 on?18:00
sdakewith modifications to load liberty versus mitaka18:00
sdakejogam i didnt tag it so i'm not ure, but i assume its on the stable/mitaka branch18:00
sdaketag = artifact produced18:00
sdakebranch = work in progress leading up to a tag18:00
Davieysdake: Sorry, are you talking about using Master kolla to deploy older OpenStack versions tan 3.0.0?18:01
sdakeDaviey no18:01
DavieyOK18:01
jogamsweet: just to recap ==> Ansible 1.9.4, docker > 1.10, git clone + checkout 2.0.1 update globals.yml to release v. 1.1.2 and then pray :)18:01
sdakeour model is one branch per release of openstack18:01
sdakeshouldn't have to pray18:01
sdakejust ping us on the channel18:01
sdakesomeone can help you out18:01
sdakeif you git checkout 2.0.1 you don't need to change globals.yml18:02
sdake(atleast for the release version :)18:02
*** salv-orlando has joined #openstack-kolla18:03
*** salv-orlando has quit IRC18:06
*** salv-orlando has joined #openstack-kolla18:06
*** haplo37 has quit IRC18:09
*** diogogmt has quit IRC18:09
*** diogogmt has joined #openstack-kolla18:10
*** g3ek has quit IRC18:10
*** haplo37 has joined #openstack-kolla18:10
*** g3ek has joined #openstack-kolla18:11
*** shardy is now known as shardy_afk18:17
*** Mr_Broken has quit IRC18:22
*** Mr_Broken has joined #openstack-kolla18:22
jogamsdake: for an eval system on VMWare, should I do: echo 'options kvm_intel nested=1' >> /etc/modprobe.d/qemu-system-x86.conf as well?18:23
sdakehrm18:23
sdakeno read the philosophy document18:23
sdakeit explains how to use qemu virt18:23
sdakekolla may or may not work on vmwre, depends on if multicast is enabled or disabled18:24
jogamsdake: did read, but also reading others using kolla and that was one of the recommendations: http://www.jinkit.com/openstack-dockerized/18:24
sdakethat may work18:25
sdakehaven't tried it myself18:25
sdakeqemu is your safest bet18:25
jogamsdake: how do you gage kolla-working, I got it starting the CirrOS image + networking for 3.0.0 from latest on Ubuntu 16.04 LTS but no way for cinder to get convinced to work, e.g. no volumes18:26
jogamyup, kvm is not working, only qemu!18:26
sdakeya nessted kvm is dessigned to run inside a parent kvm18:26
jogambut the first time I did not test the above command18:26
sdakenot inside vmware18:26
*** aNupoisc has quit IRC18:26
sdakejogam so you have kolla functional but not cinder18:27
*** ayoung has quit IRC18:27
jogamyup, was originally from kolla deploy on top of openstack18:27
sdakei.e. you ahve compute kit working18:27
jogamsdake: well kind of, now not so much anymore, I am just on my way to trash it18:27
jogamhad: 1 of each compute, ctrl, nw, and storage but ran into HD limitations for images and with Cinder into disappearing volumes18:28
jogamwhen I re-configured to Kolla deploy to have the image-registry on the storage node then images > CirrOS would time-out uploading and the show existent, but after navigation a login was required and the image was gone...18:29
jogamsdake: not sure about definition of ´compute kit´18:29
sdakefolks - this is critical - upgrades are fubared - Jeffrey4l_ can you look into working around this sproblem https://bugs.launchpad.net/kolla/+bug/159665318:31
openstackLaunchpad bug 1596653 in kolla "upgrade fails if nova vm is launched" [Critical,Confirmed]18:31
*** aNupoisc has joined #openstack-kolla18:32
openstackgerritDavid Wang proposed openstack/kolla-kubernetes: Update kolla-kubernetes installation instructions  https://review.openstack.org/33461618:32
*** Mr_Broken has quit IRC18:37
*** banix has quit IRC18:38
*** jtriley has joined #openstack-kolla18:40
*** salv-orlando has quit IRC18:40
*** Mr_Broken has joined #openstack-kolla18:44
*** b_bezak has joined #openstack-kolla18:44
inc0sdake, looks like something local to your system, can you totally clean up OS and retry?18:46
sdakei just did18:46
sdakesame reult18:46
sdakei deleted qemu-vm processes and was able to ugprade successfully18:46
sdakevm/kvm18:46
sdakelibvirt was gone after the upgrade18:47
sdakeso docker is deleting it - so it can be recreating18:47
sdakecreated18:47
sdakei think the solutoin si to ignore the error and start it again if there was one18:47
sdakeor just ignore the error18:47
sdakeyou said this works for you with aufs?18:48
inc0I'll try in a minute, but it did work with aufs18:48
sdakedid you have a vm launched?18:48
inc0yeah18:48
inc0but docker version changed in the process18:49
inc0I'll try to do it in a moment18:49
*** Mr_Broken has quit IRC18:52
*** Mr_Broken has joined #openstack-kolla18:53
*** g3ek has quit IRC18:56
*** haplo37 has quit IRC18:56
*** g3ek has joined #openstack-kolla18:56
*** haplo37 has joined #openstack-kolla18:57
openstackgerritSteven Dake proposed openstack/kolla: Use proper images when deploying mitaka  https://review.openstack.org/33458818:57
*** jtriley has quit IRC18:58
*** b_bezak has quit IRC19:02
ccesariohey guys.... does someone have any idea about this error ?  http://paste.openstack.org/show/523644/    http://paste.openstack.org/show/523645/    I need that from nova-compute container the multipathd client connect on multipathd socket running on host ... but I 'm getting coonnection refused when I try it from nova-compute container.19:02
sdakeif connection refused, must need some type of bindmount19:03
jgriffithsbezverk: ping19:04
*** sacharya_ has joined #openstack-kolla19:04
ccesariosdake, any tip ?19:04
sdakenot the faintest idea19:05
ccesariohmmm19:06
sdakepossibly something in /var/run?19:06
*** jtriley has joined #openstack-kolla19:06
sdakeit is possible it is a system level socket19:06
sdakei.e. it does not use a file to do io19:07
sdakeyou can list active sockets with netstat19:07
*** sacharya has quit IRC19:07
ccesariosdake, the socket run on /run/ (on the host)19:08
ccesariosdake, I  think that I 'm wrong. ...19:10
ccesariohttp://paste.openstack.org/show/523646/19:10
sdakeyou see that @ in front19:10
sdakethat is a system level socket19:11
sdakethat means it starts with \019:11
sdakei am not sure how to access that from inside a docker container19:11
sdakepossible multipathd can take a socket path?19:11
sdakei think the corerct term is abstract socket19:12
sdakebeen awhlie :)19:12
*** salv-orlando has joined #openstack-kolla19:12
ccesariosdake, maybe this ? http://paste.openstack.org/show/523647/19:13
sdakepossible that /var/run was my first suggetion :)19:14
sdakebut comute iirc bindmounts /run laready19:14
sdakeand /var/run symlinks to /run19:14
sdakeor visa versa19:14
ccesarioyes yes, from container I can list the multipathd socket file19:14
ccesariobut strace show me connection refused19:15
sdakewhat is strace tyring to access - filename?19:15
ccesarioconnect(3, {sa_family=AF_LOCAL, sun_path="/var/run/multipathd.sock"}, 110) = -1 ECONNREFUSED (Connection refused)19:16
sdakenot EPRM19:16
sdakeEPERM19:16
ccesariohttp://paste.openstack.org/show/523644/19:17
*** salv-orlando has quit IRC19:17
sdakecan you ls /var/run/multipathd.sock from the container?19:17
ccesarioyes19:17
ccesariohttp://paste.openstack.org/show/523645/19:17
*** berendt has joined #openstack-kolla19:20
sdakeccesario is multipathd running on your host?19:20
berendtCan we please merge https://review.openstack.org/#/c/331430/ ?19:20
patchbotberendt: patch 331430 - kolla - Add reconfigure tasks for ceilometer ansible role19:20
*** banix has joined #openstack-kolla19:20
ccesariosdake, yes19:21
sdakeinc0 that request is aimed at you ^^ :)19:21
berendtAnd can the core reviewers please check https://review.openstack.org/#/c/331573/, I do not understand why I review with the same change was merged 6 days after I opened this one...19:21
patchbotberendt: patch 331573 - kolla - Fix glance configuration templates (ABANDONED)19:21
openstackgerritKen Wronkiewicz proposed openstack/kolla-kubernetes: Adding debugging documentation for Keystone  https://review.openstack.org/33078819:21
ccesariosdake, http://paste.openstack.org/show/523648/19:21
sdakeberendt we have a large core review team different people merge different things19:22
berendtsdake do they not review open review requests first?19:22
inc0berendt, sometimes we do;) however it happened to me too that I posted a patch where similar one was already in review19:23
inc0shouldn't do it ofc19:23
*** sacharya has joined #openstack-kolla19:24
*** Mr_Broken has quit IRC19:25
sean-k-mooneyccesario: any messages from selinux/apparmor in dmesg?19:26
berendtinc0 just think that it is a waste of time to identify and fix issues two times. anyway, the issue is now fixed in the master branch :)19:26
sean-k-mooneyccesario: although maybe not as its not a permission denied19:26
ccesariosean-k-mooney, all of then disabled :)19:26
*** banix has quit IRC19:27
openstackgerritMerged openstack/kolla: Add reconfigure tasks for ceilometer ansible role  https://review.openstack.org/33143019:27
ccesariosean-k-mooney, on host of course19:27
*** salv-orlando has joined #openstack-kolla19:27
*** sacharya_ has quit IRC19:28
*** Mr_Broken has joined #openstack-kolla19:31
*** Jeffrey4l__ has joined #openstack-kolla19:32
*** Jeffrey4l_ has quit IRC19:34
*** harlowja has quit IRC19:36
*** haplo37 has quit IRC19:37
*** sdake has quit IRC19:38
*** sdake has joined #openstack-kolla19:40
sdakeberendt we have 8 pages of reviews open19:41
sdakeberendt i think we are lucky to get through 3-4 merges a day19:41
*** ayoung has joined #openstack-kolla19:42
sdakeinc0 so if i run upgrade and hat fails, and i run upgrade again it works19:43
sdakein the meantime libevirt was deleted19:44
sdakeso i think a viable workaround is just to ignore the delete failure19:44
sdakeand carry on19:44
inc0if it works19:44
inc0I'm building liberty now19:44
*** haplo37 has joined #openstack-kolla19:46
openstackgerritMerged openstack/kolla: Use proper images when deploying mitaka  https://review.openstack.org/33458819:48
openstackgerritSteven Dake proposed openstack/kolla: Liberty is now on 1.1.2 prerelease  https://review.openstack.org/33463719:49
*** b_bezak has joined #openstack-kolla19:51
*** harlowja has joined #openstack-kolla19:52
*** Mr_Broken has quit IRC19:52
*** Mr_Broken has joined #openstack-kolla19:53
sdakeinc0 i dont see where libvirt is stopped19:54
sdakein the playbooks19:54
sdakeonly that it is started19:54
*** ppowell_ has quit IRC19:54
inc0sdake, kolla_docker redeploys container if it differs19:54
inc0start.yml will redeploy container19:55
*** banix has joined #openstack-kolla19:55
ccesariosean-k-mooney, sdake suggestions about multipathd?!  :)19:57
*** b_bezak has quit IRC19:57
*** gfidente has quit IRC19:59
*** aNupoisc has quit IRC20:01
berendtsdake Hope that I can help with more reviews in the next weeks.20:05
sdakeberendt any help is appreciated :)20:06
sdakefwiw i htink most of our reviews have -1 on them for the most part20:07
*** Mr_Broken has quit IRC20:07
*** Mr_Broken has joined #openstack-kolla20:09
sdakeinc0 this works http://paste.fedoraproject.org/385669/5824914/ -> http://paste.fedoraproject.org/385669/0582491420:11
openstackgerritSteven Dake proposed openstack/kolla: Fix upgrades fail to upgrade  https://review.openstack.org/33464420:14
*** alan_ has joined #openstack-kolla20:17
*** alan_ is now known as Guest5277220:17
*** sdake_ has joined #openstack-kolla20:19
*** aernhart has quit IRC20:19
*** sdake has quit IRC20:22
inc0sdake_, that won't do good20:22
inc0that will leave liberty libvirt20:22
inc0and that's not great20:22
sdake_nah libvirt gts deleted20:22
sdake_tested manually first20:23
inc0ehh20:23
inc0no errors shoudl pass silently20:23
inc0(import this)20:23
*** Mr_Broken has quit IRC20:23
*** haplo37 has quit IRC20:23
*** g3ek has quit IRC20:23
sdake_huh20:23
*** Mr_Broken has joined #openstack-kolla20:24
sdake_the remove operation actually removes th container20:25
*** williamcaban has quit IRC20:25
sdake_however the remoe operation raises an exception in the process20:25
sdake_this causes upgrade to crater20:25
sdake_we dont want to have to run upgrade for every compute node twice20:25
sdake_its not erally an error, more like a leak warning :)20:27
*** aNupoisc has joined #openstack-kolla20:30
*** g3ek has joined #openstack-kolla20:32
*** haplo37 has joined #openstack-kolla20:33
*** jtriley has quit IRC20:35
*** Mr_Broken has quit IRC20:38
*** Mr_Broken has joined #openstack-kolla20:40
*** mliima has quit IRC20:43
*** sdake has joined #openstack-kolla20:49
openstackgerritSteven Dake proposed openstack/kolla: Liberty is now on 1.1.2 prerelease  https://review.openstack.org/33463720:51
*** sdake_ has quit IRC20:53
openstackgerritSteven Dake proposed openstack/kolla: Update version info in config file  https://review.openstack.org/33466020:53
openstackgerritSteven Dake proposed openstack/kolla: Fix upgrades fail to upgrade  https://review.openstack.org/33466120:53
*** berendt has quit IRC20:53
*** haplo37 has quit IRC21:08
openstackgerritJoshua Harlow proposed openstack/kolla: Be smarter about what to do when making a docker client  https://review.openstack.org/33017121:09
*** Mr_Broken has quit IRC21:12
*** sdake_ has joined #openstack-kolla21:12
*** Guest52772 is now known as aernhart21:12
*** sdake has quit IRC21:13
jogamsdake: nothing as stated works: when commands issued as described above (clone + checkout 2.0.1) an attempt is made to compile the latest (3.0.0 - according to docker reg) modules for centos, but I am using ubuntu!21:15
*** sdake has joined #openstack-kolla21:16
jogamcan also not find any configuration for the ´tag´ conf/cfg parameter21:16
sbezverkjogam: version tag is defined in globals.yaml21:16
*** haplo37 has joined #openstack-kolla21:17
sbezverk# Valid option is Docker repository tag21:17
sbezverkopenstack_release: "3.0.0"21:17
*** sdake_ has quit IRC21:17
jogamyup found it now, was using the /etc/kolla/globals.yml21:18
harlowjasdake inc0 got any time to explain the config stuff in kolla?21:20
harlowjahow i work, what it does :-P21:20
inc0harlowja,21:20
inc0sooo21:20
harlowjasoooo21:20
*** Mr_Broken has joined #openstack-kolla21:20
harlowjaso21:20
inc0let me get code up21:20
*** rhallisey has quit IRC21:21
harlowjai like the codes21:21
sdakeharlowja you mean ini merging?21:21
inc0https://github.com/openstack/kolla/blob/master/ansible/roles/mariadb/tasks/config.yml#L1021:21
harlowjamore the config.json stuff21:21
inc0ahh21:21
sdakeyup ive got time21:21
sdakemoment let me find a link - lets start at the top21:21
inc0config.json basically tells container which command will be called and which files should be copied from where to where21:21
harlowjalike i'm using puppet and not ansible, guess i gotta have something similar to that21:21
harlowjainc0 k21:22
inc0but basically how config in kolla works:21:22
inc0task I linked will render config and put it into kolla_config_directoryu21:22
sdakeharlowja https://github.com/openstack/kolla/blob/master/ansible/roles/heat/tasks/config.yml#L1221:22
inc0container then bindmount this directory to /var/lib/kolla21:22
sdakeharlowja this is in every ansible playbook21:22
harlowjaright21:23
sdakeinc0 he wants to see code examples21:23
harlowjai'm ok with either :-P21:23
inc0sdake, I linked it above...21:23
harlowjai take what i can get, not to picky, lol21:23
inc0so container bindmount this21:23
sdakeharlowja so ansible copies the config.json file for  the container21:23
inc0and look for config.json21:23
inc0then checks files in config.json and copies them over to /etc21:23
harlowjaright, that's via some script put into the container from kolla right?21:23
inc0so all puppet has to do is to put these files there and start container with it bindmounted to /var/lib/docker21:24
sdakeharlowja this is the json format; https://github.com/openstack/kolla/blob/master/ansible/roles/heat/templates/heat-api.json.j221:24
inc0yeah, no ansible involved aside from placing these files in host dirt21:24
inc0dir21:24
harlowjak21:24
sdakeyou can see it has the command to run and hee config files to copy around21:24
inc0so it's actually pretty easy to do the same with puppet21:24
harlowjajust format like  https://github.com/openstack/kolla/blob/master/ansible/roles/heat/templates/heat-api.json.j2 has to be made21:24
harlowja*not a template21:25
inc0yeah21:25
harlowjak21:25
sdakeharlowja for whatever container you want pretty much21:25
inc0well, at the end of the day it has to be json21:25
sdakeharlowja that json is parsed by https://github.com/openstack/kolla/blob/master/docker/base/set_configs.py21:25
*** aernhart has quit IRC21:26
harlowjaya, i gotta look at that21:26
*** aernhart has joined #openstack-kolla21:26
sdakeevery container inherits this cmd https://github.com/openstack/kolla/blob/master/docker/base/Dockerfile.j2#L26221:26
harlowjakk21:27
sdakeso when we make a heat-api container eventually it ends up running "kolla_start"21:27
harlowjayup yup, i've seen that21:27
sdakethis ends up running this script:21:27
sdakehttps://github.com/openstack/kolla/blob/master/docker/base/start.sh21:27
*** aernhart has quit IRC21:27
sdakeso every container we make runs start.sh21:27
sdakesame start.sh21:28
sdakevia inheritence from base image21:28
sdakeand inheritence of CMD operation21:28
*** aernhart has joined #openstack-kolla21:28
harlowjak21:28
sdakethis runs the aformentioned set_configs.py https://github.com/openstack/kolla/blob/master/docker/base/start.sh#L1421:28
jogamsdake: sorry but I seem to be lacking something or other, got openstack_release updated in both kolla/etc/kolla/globals.yml and in the root /etc/ folder but the compile is still generating images that are tagged as 3.0.0??21:28
jogambtw: I am compiling from 2.0.1 tag...21:28
harlowjasdake  k, cool, i'll mess around with a few things21:29
harlowjathanks inc0 sdake  for the little intro to that and what its doing21:29
sdakeset configs.py writes out the /run_command file21:29
sdakewe do it this way for security reasons21:29
sdakeso containers end up not running as root21:29
harlowjak21:29
inc0harlowja, any time, let me know if you run into any hurdles21:29
sdakein the case of a container breakout, the hacker doesnt have root privileges on the get go from an internet facing service21:29
harlowjaright, fair enough21:30
sdakethe run_command file is owned by root21:30
sdakethe set_configs.py file is owned by root21:30
sdakeothe rfiles are owned by nova or heat or whatever21:30
harlowjamakes sense21:30
sdakeso basically its not possible to escalate inside the contiainer even if you break into the contianer from a python web f acing app - which is pretty hard to begin with21:31
sdakethat said, things like /etc/nova/*21:31
sdakeare owned by the nova user21:31
sdakeso your database root pwd will be open or all to see21:31
sdakeor for the hacker that is21:31
sdakewhich is not ideal but this is just how unix permissions work21:31
harlowjaya21:31
sdakeevery other deployment needs to operate in this way21:32
sdakeopenstack could be smarter about reading the config file then dropping root21:32
sdakebut tht is a huge hurdle to tacle21:32
harlowjadef21:32
sdakemultiply that by the cargo copy paste21:32
sdake* big tent21:32
harlowjayup yup21:32
sdake= not worth my time :)21:32
sdakethis is a failure of openstack in generals security model, not of anything related to kolla21:33
sdakeany questions? :)21:34
sdakethat json file is our ABI21:34
sdakeevery container follows (for the most part) the same pattern21:34
sdakeceph doesnt drop root21:35
sdakesince its a backend service its not super significant21:35
sdakebut its not ideal - we will fix it when upstream fixes it21:35
harlowja:)21:35
harlowjathat's good enough for now21:35
harlowjagotta mess around with that more21:35
sdakewe can't make something that is intended to root drop root21:36
sdaketo run as root i mean21:36
sdakeceph is meant to run as root at th emoment21:36
sdaketheir next releae introduces root dropping21:36
openstackgerritDavid Wang proposed openstack/kolla-kubernetes: Updated quickstart doc dependencies section  https://review.openstack.org/33359121:38
sdakejogam ping21:39
sdakejogam run git status | fpate21:39
sdakefpaste if on centos21:39
sdakefi on ubuntu jsut put it in a paste service21:39
sdakeinc0 you hae any wizardry way to paste on ubuntu21:39
sdakepastebinit seems to be a favorite of ubuntu fans21:41
*** banix has quit IRC21:45
*** banix has joined #openstack-kolla21:47
*** banix has quit IRC21:48
jogamsdake: HEAD detached at 2.0.121:49
sdakepip show kolla21:50
sdakei really wish i could erwrite that documentation in a way that is more clear21:51
harlowjaalso general question, would all the stuff that is put in the base image (for centos) be ok to split up21:51
*** kproskurin has quit IRC21:51
sdakeharlowja how do you mean - please expand21:51
jogamsdake: aaahhh there is goes... Version: 3.0.0.0b2.dev16321:51
sdakejogam ya - pip remove kolla21:51
harlowjawell it seems like some virt stuff is installed even if i'm just asking for say a build of glance21:51
sdakeor maybe its uninstall21:52
sdakeone of em will get it done21:52
*** g3ek has quit IRC21:52
sdakeharlowja never seen that - but for exampe nova-compute installs libvirt21:52
harlowjaright21:52
sdakeand nova-compute doesn't need to install the 800mg of libvirt21:52
sdakei think that is the biggest offender21:53
sdakethat is really up to red hat imo - that is a packaging choice you make with rpms21:53
harlowja# kolla-build glance21:53
sdakewith source builds you get exact deps21:53
harlowjajust running that21:53
sdakeharlowja i have internal demo tomorrow i dont want  to disturb my system21:53
harlowjajust pulls in weird things :-P21:53
harlowjakk21:53
sdakeya thats rpm packaging nonsense21:54
harlowjaya, stupid rpms21:54
harlowjalol21:54
sdakethe best way to make a difference there is complain at rdo community21:54
sdakeor involve yourself there21:54
sdakei don't have time to package21:54
jogamsdake: no remove or uninstall - no files found or such, will just re-clone my virtual machine21:54
sdakeand spent 9 years of my life doing alot of packaging21:54
harlowjasdake ya, that's why i built anvil which we are getting away from21:54
sdakeso i'm done with that ;)21:54
harlowjait autopackages most of the things21:54
sdakejogam there is another approach21:54
harlowjabut ya, no thx to packages, lol21:55
sdakejogam change to whereever your python files are stored and rm -rf *kolla*21:55
sdakeas in /usr/lib/*21:55
jogamsdake: too late, she´s dead...21:56
sdakei dont use vms for dev21:58
sdakehave 5 node cluster at home and 10 node at work21:58
jogamsdake: lucky one I´d say, I am not necessarily getting paid for kolla dev and we are currently evaluating and the VMWare dual host cluster was available and unused...21:59
DavieyHey, has ceph based nova live migration failing for anyone else on Master?21:59
sdakeDaviey i'm not sure anyone has ever tried that tbh :)22:00
sdakeDaviey Jeffrey4l__ may have tried it - not sure on ceph22:01
*** g3ek has joined #openstack-kolla22:01
sdakejogam cool - well kolla is pretty cool - but for an eval it will take you probably 4-8 hours to get a hang of how it works22:01
sdakejogam but you should be able to run through the quickstart AIO in about 20 minutes22:02
sdakejogam if I follow the directoins without thinking i can run through the AIO in about 18 minutes - I timed myself at a normal pace22:02
jogamsdake: am at it since last week, got a OS cluster up and running after compiling on 16.04 LTS22:02
sdake(nto including the 15 minute build time of course;)22:02
sdakejogam thtas a real bummer - I think partof the problem is 16.0422:03
Davieysdake: i worked on Mitaka22:03
Davieyit*22:03
sdakeDaviey well thats grumpy news :(22:03
jogamsdake: true enough, but the warning signs were too small, since part of the intro´s are actually referencing 16.0422:03
sdakeDaviey mind filing a bug - sounds critical to me22:03
sdakejogam ya our docs are targeted at master22:04
Davieysdake: Yeah.. i'm investigating... I fixed two kolla/ceph bugs over the weekend22:04
sdakejogam but our code we want people to use is not master22:04
jogamhehe :)22:04
sdakeDaviey sweet - if you need reviews hit me up :)22:04
Davieysdake: This needs to land... gttps://review.openstack.org/#/c/334202/22:04
jogamsdake: I understand that, but all your browsable/googleable docu is for master :)22:04
sdakejogam i'm not quite suree how to fix it in the docs22:04
Davieyerr http*22:04
sdakejogam but i was actuallly in the midst of giving it a go right now22:05
Davieysdake: I haven't been able to reproduce the original issue that my fix reverts.22:05
sdakejogam mind i ask how many nodes your running in total22:05
jogamsdake: here is the config:http://pastebin.com/T6a6Ys7N22:09
harlowjaanyway to get a more reliable epel mirror somewhere :-P22:10
harlowjahttp://paste.openstack.org/show/523665/22:10
sdakejogam ya your running in 4 node vms, are you planning to deploy openstack only to 4 nodes22:10
jogamsdake: here globals --> http://pastebin.com/ag06Xcrk22:10
sdakeharlowja ya I know annoying22:10
sdakethat is why we can't make our gate voting22:10
harlowja:(22:10
sdakeine 20 is wrong22:10
sdakeyou want to deploy 2.0.1 if working from a tag22:11
jogamsdake: for now thats what I got resource wise - if we/the mgmt likes it it´ll be 5 med. perf. machines22:11
sdakeor 2.0.2 if working from a git repo22:11
sdakethat is stable/mitaka for example22:11
sdakecool22:11
jogamunfortunately not ´actual´ server hardware22:11
jogamsdake: not sure if to me, but all I want is to have something that works :)22:13
sdakewell you ge tthat with kolla22:13
sdakepossibly other htings like rdo22:13
sdakeor even fuel22:13
sdakei think kolla is pretty much "next generation" deployment management though22:13
sdakewe learned by  watching others with their legacy of 5+ installers :)22:14
jogamsdake: and from what I gathered online you guys are planning to release a ´more´ stable release later this year?22:14
*** huikang has quit IRC22:14
jogamsdake: that is why I am looking at kolla, since if we decide to go OS then I want something that allows me to scale out easily, vs. being a management nightmare22:14
sdakeour liberty and mitaka release was and is super stable22:15
sdakei have concerns about newton atm because we have a whole lot on our plate and we seem a bit unfoused22:15
jogam:) that sounds good, once I manage to get it somehow compiled I´ll let you know :)22:15
sdakemaster moves really fast22:16
sdakeand our testing there is insufficient22:16
sdakekeep in mind your getting upstream software22:17
sdakeand openstack in general is a little buggy here and there22:17
sdakethe stuf kolla deploys22:17
sdakeand operates22:17
wirehead_I don’t deploy OpenStack often… but when I do, I deploy master.22:17
wirehead_:D22:17
jogamsdake: to be honest I am even tempted to go back to master and ubuntu 16.04 LTS and use ceph22:18
sdakeharlowja i dont know how your rever going to pull off a bronwfield migration22:18
openstackgerritMerged openstack/kolla: Remove MariaDB warning from HAproxy  https://review.openstack.org/33405422:18
harlowjasdake thats ok, lol22:18
sdakeharlowja but if yuo manage to do so, I'll be there to help :)22:18
harlowja:-P22:18
jogamsdake: the compile and install was rather painless and it worked right away22:18
harlowjalife isn't ever simple, ha, thats ok ;)22:18
harlowjait will make a good blog post at least, lol22:18
sdakeharlowja have you evaled fuel at all22:19
harlowjanah22:19
sdakeme either22:20
harlowja:)22:20
sdakei'd like to hear it from a tech dude22:20
harlowjato much stuff to evail, lol22:20
openstackgerritJoshua Harlow proposed openstack/kolla: Consistently use sys.exit in one place  https://review.openstack.org/33469022:20
sdakewell its really hard to deploy fuel and rdo22:20
sdakehave to have real gear22:20
*** neilus has quit IRC22:21
*** inc0 has quit IRC22:21
sdakejogam line 42 is not set22:22
*** aNupoisc has quit IRC22:22
sdakejogam yet line 37 is set22:22
jogamsdake: true, makes no difference though?!22:24
sdakejogam is 100 network owned by you?22:24
sdakethis is how you wll access the services like api and whatnot22:24
jogamsdake: line 37 is easily accessible from accesible site wide22:24
sdakecool22:25
jogamsdake: you mean .10, yupp thats the idea22:25
sdakeenable_tls: yes22:25
sdakemight as well turn it on22:25
jogamsdake: but do not have tls certificates22:25
sdakeyou can generate them22:25
sdakekolla-ansible certificates22:25
jogamsdake: but they are ´fake´, no?22:25
sdakeself-signed22:25
jogamsdake: so whats the point for overhead22:26
jogamor benefit?22:26
sdakeright - they encrypt the password and whatnot22:26
sdakebut i you dont care aobut that then it doesnn't matter22:26
sdakeotherwise password goes across wire in plaintext22:26
jogamsdake: all internal network, behind double firewall and this site is really not that ´concerned´ about security it appears, like global admin passwords and the like22:27
sdakeroger22:27
sdakeif its intrnal only that sounds ok22:27
sdakei'm pretty hot on security22:28
jogamsdake: whats the performance overhead for TLS, to be honest I did not even look at it since its usually a PITA as well22:28
sdakejogam no idea on performance overhead - haven't measured it22:28
jogamsdake: try to, but not running against windmills here, got already called condescending22:28
sdakejogam the idea of openstack is you horizontally scale by adding nodes as you need22:29
*** neilus has joined #openstack-kolla22:29
sdakesomeone here?22:29
sdaketbh i think for your app you probably don't need tls22:29
sdakebut me personally - i'd always deploy tls22:30
sdakejogam did you get AIO going first?22:30
jogamsdake: that´s why I am biting my teeth on kolla22:31
jogamand openstack22:31
sdakebiting yur teeth - never heard that idiom22:31
jogamsince the site previously already decided to have to vSphere clusters (2/3 machines each) build and it is starting to become a management nightmare, now they wanted to deploy desktop dual VM machines22:31
jogamsdake: haha good catch, ESL is always going to get me...22:32
sdakei would have never known22:32
jogambut true now that I read it, interesting...22:33
sdakei have herad vsphere is easy to manage - is that not true?22:33
*** aNupoisc has joined #openstack-kolla22:33
*** tyrola has joined #openstack-kolla22:34
jogamtrue, but now you have to manage two separate clusters on two separate networks22:35
jogamplus the nightmare was going to start with the desktop VM machine ´servers´22:35
sdakehow does that even work :)22:35
sdakeyou mean remote desktop or whatever its called22:35
jogamsdake: any difference between tools/build.py + options or kolla-build22:36
sdakejogam no difference22:36
tyrolaHey guys, does anyone know what I need to configure in kolla to have gigabit network inside the guests? neutron server and compute nodes have 10gbit network cards.22:36
*** ssurana has quit IRC22:36
sdaketyrola i dont think anything22:36
openstackgerritJoshua Harlow proposed openstack/kolla: Clean all then yum update  https://review.openstack.org/33469422:36
sdaketyrola you mean you want to rate limit?22:36
tyrolasdake: Mh because I've tested multiple downloads and it seems like the ens3 device has only 100mbit/s22:37
tyrolaethtool isn't very helpful22:37
sdakeiv'e never tried benchmarkign on my 10gig switch22:37
sdakei know it makes a world of difference in deploys22:38
*** ssurana has joined #openstack-kolla22:38
sdakeand has a big impact on ceph bandwidth22:38
sdakei've actuaally benchmarked ceph22:38
sdakeits near wire speed22:38
sdakeif yo utake into account all the crap it needs to do to ensure reduandacy22:39
sdakeas to your particular question22:39
Davieysdake: hey, would you mind running a command for me on a compute node?22:39
Daviey$ sudo docker exec -ti nova_libvirt /bin/bash22:39
Daviey(nova-libvirt)[root@amcbk-qb161201 /]# virsh dumpxml $(virsh list | grep instance | head -n1 | awk '{ print $1 }') | grep vnc22:39
sdakeDaviey i have mitaka deployed22:39
Davieyperfect22:39
sdakeok moment22:39
DavieyI want to compare mitaka with newton22:39
jogamsdake: no AIO - four nodes right away22:40
*** haplo37 has quit IRC22:40
*** g3ek has quit IRC22:40
*** Mr_Broken has quit IRC22:40
jogamsdake: VM desktops are supposed to host identical VM images accross 5/6 machines that in case of destruction by user we were supposed to ´recover´ from USB HD22:41
*** Mr_Broken has joined #openstack-kolla22:41
Davieysdake: I need to step away, but i will read scrollback.  Ta22:41
sdakeDaviey     <graphics type='vnc' port='5900' autoport='yes' listen='192.168.1.103' keymap='en-us'>22:41
tyrolasdake: I think it can be really a issue with my 3par storage (I don't use ceph). Its integrated via iscsi. I've tested with this test file "wget http://speedtest.tele2.net/1000GB.zip -O /dev/null". If I use iperf it seems to be much faster.22:42
openstackgerritJoshua Harlow proposed openstack/kolla: Clean all then yum update  https://review.openstack.org/33469422:43
sdaketyrola are you using master?22:43
tyrolasdake: nope, the mitaka branch but I've merged some icsci stuff 2-3 weeks ago from the master to the branch22:44
sdakenice22:44
*** aNupoisc has quit IRC22:45
sdaketyrola mind providing top 3 pain points of kolla :)22:45
sdaketyrola we are really trying to learn from operators that use our software22:45
*** ssurana has quit IRC22:45
sdaketyrola isdsi is going to be super slow in generlal because it introduces extra copies on the wire22:46
tyrolasdake: I hope I can help you... Maybe there is some kolla issue why the iscsi? does not provide more then 100mbit/s22:46
sdakewell ceph for example is fullly meshed22:47
sdakemeaning every node can access storage directly from where it needs it22:47
tyrolaEach compute node has a dedicated 10gbit port for icsci in a seperate vlan, but sure maybe its too slow.22:47
sdakeiscsi is single connection to the storage backend22:47
*** g3ek has joined #openstack-kolla22:48
sdakeits just the way the technology works22:48
sdakebut i think its worth investigating further22:48
*** haplo37 has joined #openstack-kolla22:48
sdakeif iscsi was mae to work in a mesh, then it would work much like ceph22:48
sdakebut thatis a big job22:49
sdakeas in 2-3 years of effort22:49
tyrolaOh yes, change our storage stuff to fibrechannel would be very nice but also very expensive. We bought the HP 3par just a few weeks ago...22:49
sdakeceph can be used with cots jbod arrays22:49
sdakeno need for fibrechnanel22:49
sdakeuse copper on 10gig22:49
sdakebut you bought it so i get it :)22:50
jogamsdake: binary vs source any preference22:50
sdakejogam i prefer either on centos22:50
sdakesource definately on ubuntu22:50
sdakesource makes smaller images22:51
sdakebut changes more often22:51
sdakebinary ubuntu is busted atm22:51
jogamsdake: oh ya I remember that was my first compile...22:51
jogamsdake: you suggesting that ubuntu is not likable vs. centos?22:52
sdakepersoanlly if you want latest and greatest i'd go with centos22:52
jogamsdake: really... hm!22:52
sdakeubutnu 16.04 will be latet and greatest but the reality is we are 3 months away from an implentation22:52
sdakeand that will be newton22:52
sdakered hat has bigger r&d team to implement rdo and centos as well22:53
jogamsdake: true, but my personal experience with red has been pita22:53
sdakeyup - i use centos daily ad like it more then i did the fedora experience22:53
sdakebut i know ho wto use rpm based distros22:54
sdakebut dont know how to use apt based distros22:54
sdakeso sounds like you know how to use apt bseed distros so i'd go with ource :)22:54
jogamsdake: apt vs rpm is only a difference of commands no?22:55
sdakethe whole distro is laid out differently22:55
sdakei've used fedora for 10+ years22:55
sdakei am intimately familiar with everything about it22:55
jogammy experience is more on the side when something is not working its a lot easier to fix in Debian/Ubuntu than the red(s)22:55
sdakeand centos comes from fedora22:55
sdakewell systemd has my whole systme in lockdown which I don't like22:56
openstackgerritDavid Wang proposed openstack/kolla-kubernetes: Update kolla-kubernetes installation instructions  https://review.openstack.org/33461622:57
sdakebut 16.04 has systemd22:57
*** Mr_Broken has quit IRC22:57
jogamsdake: probably wrong perspectives, mine is as a user trying to find a set of commands to fix issue X and that seems to be being a lot better solved for Ubuntu than others...22:57
jogamsdake: had the honor to ´work´ with Oracle Linux for a couple of days here, oh my, lost a lot of hair...22:58
*** salv-orl_ has joined #openstack-kolla22:59
sdakeone thing aout fedora - every 6 months it was upgrade time22:59
sdakeand that was painful22:59
sdakebut you allways were rolling with the latest and greatest22:59
sdakei saw the whole systemd thing evolve over 5 releaswes22:59
jogamhm, my experience with Fedora is mixed/scewed only had to maintain a Fed.5 server for a process control system and that was really painful23:01
*** salv-orlando has quit IRC23:01
jogamwhat makes systemd so unappetizing for your23:02
jogam-r23:02
*** vhosakot has quit IRC23:03
openstackgerritJoshua Harlow proposed openstack/kolla: Allow for externally managed configuration  https://review.openstack.org/33259023:03
openstackgerritJoshua Harlow proposed openstack/kolla: Allow for externally managed configuration  https://review.openstack.org/33259023:05
sdakesystemd is fine23:06
sdakeuefi hs been an adjustment23:06
sdakethe bios the way it was was fine23:06
tyrolasdake: do you think there is a config issue with the libvirt iscsi driver? I've tested the guest networking performance now with different iperf public servers. Normal network seems to be okay (gigabit works). But icsci seems to be limited to 100mbit/s23:07
sdaketyrola is it exactly 100mbit?23:07
tyrola110~23:07
sdakesbezverk around?23:07
*** Mr_Broken has joined #openstack-kolla23:09
tyrolaI think I will try tomorrow a higher mtu.23:12
tyrolaI've checked a document from HP http://h20564.www2.hpe.com/hpsc/doc/public/display?docId=emr_na-c04873990 and they recommend MTU 9000 (jumbo frames)23:12
sdakeoh ya jumbo frames area a must23:12
sdakethat will make a big improvement23:13
sdakelet me see if i can figure how to turn that on23:13
tyrolaThat would be nice, I will check if we need to change some settings on the 3par and on the switches23:14
tyrolaBut our networking admin is already out of office, I think I need to wait for tomorrow then23:14
sdakeya you will need to enable umbo frame sbefore we can experiment with getting that going in your openstack deployment23:16
sdakei can telly ou its possible23:16
sdakeits documented: http://docs.openstack.org/juno/install-guide/install/apt-debian/content/neutron-network-node.html23:16
sdakewhat I can't tell you is if it will improve performance23:16
*** aNupoisc has joined #openstack-kolla23:17
sdaketyrola mind i sk how many nodes you have23:18
tyrolacurrently, 2x controller, 2x networking nodes (neutron) and 4x compute nodes23:18
tyrolathe networking and compute nodes does have 2x 10gbit ports (neutron, one internal one external interface) and the compute nodes (storage and network interface)23:19
jogamseem tag 2.0.1 has a couple of ubuntu source compile targets that fail --> http://paste.openstack.org/show/523672/23:19
tyrolasdake: if you need I can send you full specs, we can continue talking tomorrow if we changed our setup to jumbo frames23:21
jogamtried to recompile mesosphere-base and got keyserver.ubuntu.com timeout --> http://paste.openstack.org/show/523673/23:22
*** Mr_Broken has quit IRC23:24
*** Mr_Broken has joined #openstack-kolla23:25
*** haplo37 has quit IRC23:25
*** jtriley has joined #openstack-kolla23:29
*** haplo37 has joined #openstack-kolla23:34
*** Mr_Broken has quit IRC23:38
*** sacharya has quit IRC23:40
*** tyrola has quit IRC23:41
*** Mr_Broken has joined #openstack-kolla23:41
*** tyrola has joined #openstack-kolla23:42
*** aernhart has quit IRC23:46
*** aernhart has joined #openstack-kolla23:48
*** Mr_Broken has quit IRC23:54
*** Mr_Broken has joined #openstack-kolla23:54
*** tyrola has quit IRC23:55

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!