| *** v1k0d3n has joined #openstack-kolla | 00:13 | |
| *** Mr_Broken has joined #openstack-kolla | 00:38 | |
| *** Mr_Broken has quit IRC | 00:42 | |
| *** daneyon has joined #openstack-kolla | 00:45 | |
| *** daneyon has quit IRC | 00:50 | |
| *** dwalsh has quit IRC | 00:51 | |
| openstackgerrit | Jack Ning proposed openstack/kolla: Replace horizon default config with custom config https://review.openstack.org/306928 | 00:55 |
|---|---|---|
| *** zhurong has joined #openstack-kolla | 01:16 | |
| *** salv-orl_ has quit IRC | 01:25 | |
| *** phuongnh has joined #openstack-kolla | 01:29 | |
| *** g3ek has quit IRC | 01:40 | |
| *** haplo37 has quit IRC | 01:40 | |
| *** zhiwei has joined #openstack-kolla | 01:45 | |
| *** haplo37 has joined #openstack-kolla | 01:47 | |
| *** g3ek has joined #openstack-kolla | 01:48 | |
| *** JianqingJiang has joined #openstack-kolla | 01:51 | |
| *** salv-orlando has joined #openstack-kolla | 01:56 | |
| *** hanchao has joined #openstack-kolla | 01:56 | |
| openstackgerrit | Britt Houser proposed openstack/kolla: Fix minor typo in security guide. https://review.openstack.org/334272 | 02:04 |
| *** salv-orlando has quit IRC | 02:05 | |
| openstackgerrit | Britt Houser proposed openstack/kolla: [TrivialFix] Typo in security guide. https://review.openstack.org/334272 | 02:23 |
| *** sdake_ has quit IRC | 02:23 | |
| *** Mr_Broken has joined #openstack-kolla | 02:26 | |
| *** Mr_Broken has quit IRC | 02:31 | |
| openstackgerrit | Britt Houser proposed openstack/kolla: Fix minor typo in security guide. https://review.openstack.org/334272 | 02:32 |
| *** yuanying has quit IRC | 02:49 | |
| *** klint has joined #openstack-kolla | 02:49 | |
| *** salv-orlando has joined #openstack-kolla | 03:02 | |
| openstackgerrit | Md Nadeem proposed openstack/kolla: Fix container stop exeption https://review.openstack.org/333744 | 03:03 |
| *** salv-orlando has quit IRC | 03:10 | |
| openstackgerrit | Merged openstack/kolla: Fix minor typo in security guide. https://review.openstack.org/334272 | 03:12 |
| openstackgerrit | Merged openstack/kolla: glance-*, fix rabbit config with ceilometer https://review.openstack.org/334205 | 03:13 |
| openstackgerrit | Merged openstack/kolla: Fix docker daemon proxy support in vagrant https://review.openstack.org/333238 | 03:17 |
| *** coolsvap has joined #openstack-kolla | 03:17 | |
| *** Mr_Broken has joined #openstack-kolla | 03:20 | |
| *** v1k0d3n has quit IRC | 03:20 | |
| *** Jeffrey4l_ has quit IRC | 03:23 | |
| *** robcresswell has quit IRC | 03:24 | |
| *** Mr_Broken has quit IRC | 03:25 | |
| *** Daviey has quit IRC | 03:25 | |
| *** Daviey has joined #openstack-kolla | 03:27 | |
| *** robcresswell has joined #openstack-kolla | 03:27 | |
| *** Jeffrey4l_ has joined #openstack-kolla | 03:36 | |
| *** sacharya has joined #openstack-kolla | 03:41 | |
| *** yuanying has joined #openstack-kolla | 03:47 | |
| openstackgerrit | Hui Kang proposed openstack/kolla: Enable openvswitch container logs in host volumes https://review.openstack.org/334283 | 03:49 |
| mdnadeem | coolsvap, ping | 03:59 |
| *** Jeffrey4l_ has quit IRC | 04:00 | |
| hanchao | hi guys, have you suffered with the unstable rabbitmq image for Liberty(1.1.0)? | 04:01 |
| *** stvnoyes has quit IRC | 04:01 | |
| *** stvnoyes has joined #openstack-kolla | 04:02 | |
| mdnadeem | hanchao, No idea for v 1.1.0, however i have build it on Liberrty(1.1.1) Centos successfully | 04:04 |
| hanchao | the image itself can be built successfully. but when i deploy that, it's really unstable... don't know what's your experience? | 04:06 |
| *** salv-orlando has joined #openstack-kolla | 04:07 | |
| mdnadeem | hanchao, i have deployedon v 1.1.1 successfully | 04:08 |
| *** Jeffrey4l_ has joined #openstack-kolla | 04:09 | |
| hanchao | and your openstack cluster works well? | 04:09 |
| hanchao | if so, maybe i should try that. | 04:09 |
| *** gbraad has joined #openstack-kolla | 04:11 | |
| mdnadeem | hanchao, yes, thats work fine | 04:12 |
| hanchao | mdnadeem: btw, and did you built all images by this tag? or only for the rabbitmq? | 04:13 |
| mdnadeem | hanchao, all image | 04:13 |
| hanchao | I'll try it, thx :) | 04:15 |
| *** salv-orlando has quit IRC | 04:15 | |
| coolsvap | mdnadeem, pong whatsup? | 04:21 |
| *** Jeffrey4l_ has quit IRC | 04:21 | |
| mdnadeem | coolsvap, hi | 04:24 |
| mdnadeem | coolsvap, my haproxy container restart because of nova-novnc proxy, haproxy tries to bind socket [192.168.122.60:6080] | 04:24 |
| hanchao | I proposed a suggestion if anyone could have a look at and leave comments [https://review.openstack.org/#/c/333666/ ], thx in advance. | 04:25 |
| patchbot | hanchao: patch 333666 - kolla - Add the verification of required images step befor... | 04:25 |
| mdnadeem | however, it haproxy container log show : Running command: '/usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid' | 04:25 |
| mdnadeem | [WARNING] 178/031940 (8) : config : 'option forwardfor' ignored for proxy 'mariadb' as it requires HTTP mode. | 04:25 |
| mdnadeem | [ALERT] 178/031940 (8) : Starting proxy nova_novncproxy: cannot bind socket [192.168.122.60:6080] | 04:25 |
| mdnadeem | coolsvap, nova-novnc service already bind this port to 0.0.0.0, although in nova.conf novncproxy_host = 192.168.122.185 is defined | 04:27 |
| mdnadeem | I am worried why nova-novncproxy service bind to 0.0.0.0 even when novncproxy_host is defined in nova.conf ? | 04:27 |
| coolsvap | mdnadeem, i dont have a multinode setup to test and verify tbh but i will give a try | 04:28 |
| mdnadeem | coolsvap, ohk, thanks | 04:29 |
| *** haplo37 has quit IRC | 04:34 | |
| *** Jeffrey4l_ has joined #openstack-kolla | 04:39 | |
| *** salv-orlando has joined #openstack-kolla | 04:42 | |
| *** tfukushima has joined #openstack-kolla | 04:42 | |
| *** haplo37 has joined #openstack-kolla | 04:42 | |
| openstackgerrit | Merged openstack/kolla: Fix container stop exeption https://review.openstack.org/333744 | 04:45 |
| *** salv-orlando has quit IRC | 04:50 | |
| *** daneyon has joined #openstack-kolla | 05:16 | |
| *** tfukushima has quit IRC | 05:19 | |
| *** g3ek has quit IRC | 05:20 | |
| *** daneyon has quit IRC | 05:21 | |
| *** haplo37 has quit IRC | 05:21 | |
| *** g3ek has joined #openstack-kolla | 05:26 | |
| *** haplo37 has joined #openstack-kolla | 05:27 | |
| *** salv-orlando has joined #openstack-kolla | 05:38 | |
| *** fragatina has joined #openstack-kolla | 05:40 | |
| *** fragatina has quit IRC | 05:40 | |
| *** fragatina has joined #openstack-kolla | 05:41 | |
| *** tfukushima has joined #openstack-kolla | 05:49 | |
| *** JeroenBo has joined #openstack-kolla | 05:59 | |
| *** diga has joined #openstack-kolla | 06:02 | |
| *** haplo37 has quit IRC | 06:02 | |
| *** Mr_Broken has joined #openstack-kolla | 06:02 | |
| *** g3ek has quit IRC | 06:04 | |
| *** g3ek has joined #openstack-kolla | 06:04 | |
| *** haplo37 has joined #openstack-kolla | 06:05 | |
| *** bootsha has joined #openstack-kolla | 06:05 | |
| *** Mr_Broken has quit IRC | 06:07 | |
| *** neilus has joined #openstack-kolla | 06:19 | |
| *** v1k0d3n has joined #openstack-kolla | 06:35 | |
| *** v1k0d3n has quit IRC | 06:40 | |
| *** g3ek has quit IRC | 06:42 | |
| *** haplo37 has quit IRC | 06:46 | |
| openstackgerrit | Merged openstack/kolla: Stop using a global logger for all the things https://review.openstack.org/321884 | 06:48 |
| *** haplo37 has joined #openstack-kolla | 06:48 | |
| openstackgerrit | Mohammed Salih Puthenpurayil proposed openstack/kolla: Trivial fixes to correct couple of typos. https://review.openstack.org/334306 | 06:50 |
| *** g3ek has joined #openstack-kolla | 06:51 | |
| *** Mr_Broken has joined #openstack-kolla | 06:53 | |
| *** Serlex has joined #openstack-kolla | 06:54 | |
| *** sacharya has quit IRC | 06:55 | |
| *** Mr_Broken has quit IRC | 07:05 | |
| *** Mr_Broken has joined #openstack-kolla | 07:08 | |
| *** belmoreira has joined #openstack-kolla | 07:09 | |
| *** Mr_Broke_ has joined #openstack-kolla | 07:21 | |
| *** Mr_Broken has quit IRC | 07:23 | |
| *** Mr_Broken has joined #openstack-kolla | 07:27 | |
| *** Mr_Broke_ has quit IRC | 07:29 | |
| *** dmk0202 has joined #openstack-kolla | 07:31 | |
| *** Mr_Broken has quit IRC | 07:31 | |
| *** Mr_Broken has joined #openstack-kolla | 07:31 | |
| *** Mr_Broken has quit IRC | 07:37 | |
| *** Mr_Broken has joined #openstack-kolla | 07:41 | |
| *** jmccarthy has left #openstack-kolla | 07:42 | |
| *** shardy has joined #openstack-kolla | 07:47 | |
| *** Mr_Broken has quit IRC | 07:53 | |
| *** sacharya has joined #openstack-kolla | 07:56 | |
| *** Mr_Broken has joined #openstack-kolla | 07:58 | |
| *** b_bezak has joined #openstack-kolla | 08:00 | |
| *** Mr_Broke_ has joined #openstack-kolla | 08:01 | |
| *** sacharya has quit IRC | 08:01 | |
| *** athomas has joined #openstack-kolla | 08:03 | |
| *** Mr_Brok__ has joined #openstack-kolla | 08:03 | |
| *** neilus has quit IRC | 08:03 | |
| *** Mr_Broken has quit IRC | 08:03 | |
| *** salv-orlando has quit IRC | 08:04 | |
| *** Mr_Broke_ has quit IRC | 08:06 | |
| *** neilus has joined #openstack-kolla | 08:06 | |
| *** Mr_Broken has joined #openstack-kolla | 08:08 | |
| *** Mr_Brok__ has quit IRC | 08:09 | |
| *** Mr_Broke_ has joined #openstack-kolla | 08:11 | |
| *** mewald has joined #openstack-kolla | 08:13 | |
| *** Mr_Broken has quit IRC | 08:13 | |
| *** g3ek has quit IRC | 08:15 | |
| gbraad | .join #bifrost | 08:15 |
| *** haplo37 has quit IRC | 08:16 | |
| *** gbraad has quit IRC | 08:16 | |
| *** gbraad has joined #openstack-kolla | 08:16 | |
| *** Mr_Broken has joined #openstack-kolla | 08:19 | |
| *** Mr_Broke_ has quit IRC | 08:21 | |
| *** haplo37 has joined #openstack-kolla | 08:21 | |
| *** g3ek has joined #openstack-kolla | 08:21 | |
| *** Mr_Broken has quit IRC | 08:23 | |
| *** salv-orlando has joined #openstack-kolla | 08:25 | |
| *** dwalsh has joined #openstack-kolla | 08:27 | |
| *** shardy has quit IRC | 08:32 | |
| *** shardy has joined #openstack-kolla | 08:34 | |
| *** Mr_Broken has joined #openstack-kolla | 08:34 | |
| *** phuongnh has quit IRC | 08:34 | |
| *** Mr_Broke_ has joined #openstack-kolla | 08:38 | |
| *** Mr_Broken has quit IRC | 08:39 | |
| openstackgerrit | Md Nadeem proposed openstack/kolla: Add unit test for stop and restart container https://review.openstack.org/331524 | 08:42 |
| *** Mr_Broken has joined #openstack-kolla | 08:45 | |
| *** Mr_Broke_ has quit IRC | 08:46 | |
| *** Mr_Broken has quit IRC | 08:50 | |
| *** Mr_Broken has joined #openstack-kolla | 08:51 | |
| *** dwalsh has quit IRC | 08:51 | |
| *** tfukushima has quit IRC | 08:52 | |
| *** tfukushima has joined #openstack-kolla | 08:53 | |
| *** sacharya has joined #openstack-kolla | 08:57 | |
| *** Mr_Broken has quit IRC | 08:58 | |
| *** g3ek has quit IRC | 08:59 | |
| *** Mr_Broken has joined #openstack-kolla | 08:59 | |
| *** haplo37 has quit IRC | 09:00 | |
| *** neilus has quit IRC | 09:00 | |
| *** haplo37 has joined #openstack-kolla | 09:01 | |
| *** neilus has joined #openstack-kolla | 09:01 | |
| *** g3ek has joined #openstack-kolla | 09:01 | |
| *** sacharya has quit IRC | 09:01 | |
| openstackgerrit | Mohammed Salih Puthenpurayil proposed openstack/kolla: Trivial fixes to correct couple of typos. https://review.openstack.org/334306 | 09:03 |
| JianqingJiang | Can Kolla achieve high availability? | 09:05 |
| *** Mr_Broke_ has joined #openstack-kolla | 09:07 | |
| *** Mr_Broken has quit IRC | 09:09 | |
| sean-k-mooney | kolla supprots ha deployment of most if not all the services it can deploy | 09:11 |
| sean-k-mooney | achiveing high availablity is rathaer subjective though | 09:11 |
| *** Mr_Broken has joined #openstack-kolla | 09:14 | |
| *** Mr_Broke_ has quit IRC | 09:16 | |
| *** Mr_Broke_ has joined #openstack-kolla | 09:16 | |
| *** Mr_Broken has quit IRC | 09:18 | |
| *** Mr_Broken has joined #openstack-kolla | 09:18 | |
| JianqingJiang | OK.But I think if there is a config file to customize HA option is better | 09:20 |
| *** Mr_Broke_ has quit IRC | 09:20 | |
| *** Mr_Broken has quit IRC | 09:23 | |
| *** Mr_Broken has joined #openstack-kolla | 09:28 | |
| *** Mr_Broke_ has joined #openstack-kolla | 09:31 | |
| *** Mr_Broken has quit IRC | 09:32 | |
| *** Mr_Broken has joined #openstack-kolla | 09:33 | |
| *** Mr_Broke_ has quit IRC | 09:35 | |
| *** Mr_Broken has quit IRC | 09:37 | |
| *** Mr_Broken has joined #openstack-kolla | 09:38 | |
| sean-k-mooney | JianqingJiang: you can customise the ha deployment via the ansible inventory file. | 09:40 |
| *** g3ek has quit IRC | 09:43 | |
| *** Mr_Broke_ has joined #openstack-kolla | 09:43 | |
| JianqingJiang | Is there some documents about HA deployment via ansible inventory file? | 09:44 |
| *** prithiv has joined #openstack-kolla | 09:44 | |
| pbourke | JianqingJiang: http://docs.openstack.org/developer/kolla/multinode.html | 09:44 |
| *** haplo37 has quit IRC | 09:44 | |
| sean-k-mooney | kolla is ha by defult if you are not using singel node | 09:45 |
| *** Mr_Broken has quit IRC | 09:45 | |
| *** haplo37 has joined #openstack-kolla | 09:45 | |
| *** zhiwei has quit IRC | 09:46 | |
| *** g3ek has joined #openstack-kolla | 09:46 | |
| *** bootsha has quit IRC | 09:47 | |
| *** neilus has quit IRC | 09:47 | |
| *** neilus has joined #openstack-kolla | 09:48 | |
| *** neilus has quit IRC | 09:48 | |
| *** neilus has joined #openstack-kolla | 09:49 | |
| *** neilus has quit IRC | 09:49 | |
| *** neilus has joined #openstack-kolla | 09:50 | |
| pbourke | correct | 09:50 |
| *** mewald has quit IRC | 09:51 | |
| *** tyrola has joined #openstack-kolla | 09:52 | |
| JianqingJiang | But I notice that every service is only one container to provide | 09:52 |
| *** prithiv has quit IRC | 09:55 | |
| *** Mr_Broken has joined #openstack-kolla | 09:55 | |
| *** strigazi_ is now known as strigazi | 09:56 | |
| *** Mr_Broke_ has quit IRC | 09:57 | |
| *** zhurong has quit IRC | 10:00 | |
| *** Mr_Broke_ has joined #openstack-kolla | 10:02 | |
| sean-k-mooney | JianqingJiang: can you expand on that? | 10:02 |
| sean-k-mooney | JianqingJiang: each container runs a singel service. ha deployment are achived by running | 10:02 |
| *** Mr_Broken has quit IRC | 10:02 | |
| sean-k-mooney | JianqingJiang: several instances of the same container on multiple hosts | 10:03 |
| sean-k-mooney | JianqingJiang: haporxy and keepalived are then used to loadblance across the api services | 10:03 |
| sean-k-mooney | JianqingJiang: mariadb is clusered using glara if i recall and i think Rabbit MQs native clustering is also used. | 10:04 |
| JianqingJiang | That is for stateless service,but how to achieve ha about stateful service? | 10:04 |
| JianqingJiang | OK | 10:04 |
| JianqingJiang | Thanks a lot | 10:05 |
| *** neilus has quit IRC | 10:07 | |
| *** Mr_Broken has joined #openstack-kolla | 10:08 | |
| *** Mr_Broke_ has quit IRC | 10:09 | |
| *** b_bezak has quit IRC | 10:12 | |
| *** b_bezak has joined #openstack-kolla | 10:13 | |
| *** mewald has joined #openstack-kolla | 10:13 | |
| *** Mr_Broke_ has joined #openstack-kolla | 10:13 | |
| *** Mr_Broken has quit IRC | 10:14 | |
| *** dims has quit IRC | 10:14 | |
| *** Mr_Broken has joined #openstack-kolla | 10:16 | |
| *** Mr_Broke_ has quit IRC | 10:19 | |
| *** dims has joined #openstack-kolla | 10:20 | |
| *** JianqingJiang has quit IRC | 10:24 | |
| *** Mr_Broken has quit IRC | 10:24 | |
| tyrola | Hey guys, I am fighting with neutron openvswitch in our kolla setup now for 2 days. I've running a multinode cluster with kolla on cent os 7. But the router gateway address and the floating ips are not reachable. I can't find the reason. I also see no errors in the whole log files. | 10:25 |
| *** bootsha has joined #openstack-kolla | 10:26 | |
| *** Mr_Broken has joined #openstack-kolla | 10:26 | |
| *** neilus has joined #openstack-kolla | 10:28 | |
| sean-k-mooney | tyrola: did you create the router manually or with the init_runonce script? | 10:28 |
| tyrola | manually, tried via gui and cli | 10:29 |
| sean-k-mooney | on your network node can you see the network namespace of the router? | 10:30 |
| tyrola | http://paste.openstack.org/show/AzG6QJJDnZgmuW02k7bq/ (commands I've used) | 10:30 |
| *** tfukushima has quit IRC | 10:32 | |
| tyrola | Yap, just on the 2nd node. One node has only one for dhcp and one has 2 (dhcp and router) | 10:32 |
| sean-k-mooney | ok the command look correct but i belive the uplink port on the will not have the gateway ip so you will have assign 5.83.160.1 to the br-ex on the node that has qouter namespace | 10:33 |
| tyrola | and the router namespace has 3 interfaces, lo, one for internal ips, and one floating ip interface | 10:34 |
| tyrola | The subnet has already a external router from our datacenter with ip 5.83.160.1 | 10:34 |
| tyrola | this router is maintained by our carrier | 10:35 |
| sean-k-mooney | ok yes that should work too. am from the router network namespace can you ping that upstream router ip? | 10:35 |
| sean-k-mooney | sudo ip netns exec <route ns> ping <upstream router> | 10:36 |
| tyrola | Okay, not that isn't working.. maybe the bridges aren't correct? | 10:36 |
| tyrola | In which state must be each bridge on a working setup? (br-int, br-tun and br-ex)? | 10:37 |
| sean-k-mooney | i assume you added an interface directly to the br-ex bridge instead of nating it with iptables | 10:38 |
| sean-k-mooney | the local ports on the br-* interfaces do not need to be up | 10:38 |
| *** prithiv has joined #openstack-kolla | 10:38 | |
| sean-k-mooney | you would only bringe them up for debugging | 10:38 |
| tyrola | Does kolla add the external interface normally to br-ex? | 10:39 |
| sean-k-mooney | the interfaces that are directly added to the br-ex do need to be brought up in the kernel | 10:39 |
| sean-k-mooney | no | 10:39 |
| tyrola | Okay.. that should be already the reason. damn | 10:39 |
| sean-k-mooney | well | 10:39 |
| sean-k-mooney | kolla will a an interface that will be used by the vm data network to the br-ex | 10:40 |
| sean-k-mooney | is should have been more clear | 10:40 |
| sean-k-mooney | that interface can be used for upstream internet traffic also | 10:40 |
| sean-k-mooney | kolla will not change the interface state however | 10:40 |
| sean-k-mooney | it will jsut add it to the ovs bridge | 10:41 |
| sean-k-mooney | can you ifcofnig on the interface that is added to the br-ex and check that it is up | 10:41 |
| tyrola | I will test it and give you feedback in a few minutes :) Thanks | 10:42 |
| *** daneyon has joined #openstack-kolla | 10:42 | |
| *** daneyon has quit IRC | 10:46 | |
| *** gfidente has joined #openstack-kolla | 10:47 | |
| *** fragatina has quit IRC | 10:48 | |
| *** fragatina has joined #openstack-kolla | 10:49 | |
| *** mewald has quit IRC | 10:49 | |
| tyrola | It seems like the br-XX interfaces on the neutron nodes aren't bridges | 10:53 |
| tyrola | brctl show doesn't shows them up correctly. what the best and correct way to configure the network? Can't find anything in the kolla documentation for the bridge setup. | 10:54 |
| tyrola | But if I run inside the openvswitch container "ovs-vsctl show" it shows up the br-ex interface with Port external | 10:56 |
| sean-k-mooney | the bridge should not be shown in brctl you should use "ovs-vsctl show" to view the ovs bridges | 10:56 |
| sean-k-mooney | yes | 10:56 |
| sean-k-mooney | you will need to be in the ovs container to see the bridge config | 10:56 |
| sean-k-mooney | when you look at the br-ex you should see the interface specified in neutron_external_interface: "eth4" added to the bridge | 10:57 |
| *** sacharya has joined #openstack-kolla | 10:58 | |
| tyrola | http://paste.openstack.org/show/hX3yiMl48kiF9EC7u3Y7/ full output | 10:58 |
| *** salv-orl_ has joined #openstack-kolla | 10:59 | |
| sean-k-mooney | i take it you set neutron_external_interface: external in the global.yml | 11:00 |
| tyrola | yap thats correct, the full interfaces list http://paste.openstack.org/show/a9BTlh0xt4HHnAZGkKeE/ | 11:00 |
| *** salv-orl_ has quit IRC | 11:01 | |
| *** salv-orlando has quit IRC | 11:02 | |
| *** sacharya has quit IRC | 11:02 | |
| sean-k-mooney | hum the interface appears to be up. so looking at the vsctl out put you should have a flow to strip vlan tag 4 and output normal on br-ex | 11:04 |
| sean-k-mooney | sudo ovs-ofctl dump-flow br-ex | 11:05 |
| sean-k-mooney | you should also have a reverse flow that will add vlan4 if traffic coming in port int-br-ex untagged on the br-int | 11:06 |
| sean-k-mooney | are they present? | 11:06 |
| *** Mr_Broken has quit IRC | 11:07 | |
| *** tyrola has quit IRC | 11:08 | |
| *** tyrola has joined #openstack-kolla | 11:09 | |
| tyrola | http://paste.openstack.org/show/242g0hxd3lOBHcjqVoZf/ | 11:09 |
| tyrola | Why vlan tag 4? where is this set? | 11:11 |
| sean-k-mooney | Port "qg-2d69639a-b5" | 11:12 |
| sean-k-mooney | tag: 4 | 11:12 |
| sean-k-mooney | Interface "qg-2d69639a-b5" | 11:12 |
| sean-k-mooney | qg => quantum(old name for neutron) gateway | 11:12 |
| tyrola | Must the tag id match with the external network vlan tag id? or is this separated with the namespace? | 11:13 |
| sean-k-mooney | the tag id is purly internal for tenent isolation | 11:13 |
| tyrola | Ok alright | 11:13 |
| sean-k-mooney | neutron will translate it to the appropriate external vlan | 11:13 |
| sean-k-mooney | in this case you created a flat network | 11:13 |
| sean-k-mooney | so no vlan is used | 11:13 |
| *** Mr_Broken has joined #openstack-kolla | 11:14 | |
| *** g3ek has quit IRC | 11:14 | |
| *** haplo37 has quit IRC | 11:14 | |
| sean-k-mooney | looking at the flow on br-ex they are correctly set up | 11:14 |
| sean-k-mooney | cookie=0x92695e18ab45a9ea, duration=2748.342s, table=0, n_packets=73, n_bytes=3066, idle_age=288, priority=4,in_port=2,dl_vlan=4 actions=strip_vlan,NORMAL | 11:15 |
| sean-k-mooney | this flow is being hit | 11:15 |
| sean-k-mooney | so traffic should be getting to your phyical network untagged | 11:15 |
| *** haplo37 has joined #openstack-kolla | 11:16 | |
| *** g3ek has joined #openstack-kolla | 11:16 | |
| sean-k-mooney | if you ran tcpdump -i external and ping the datacenter router form the neutron router namespace you should see the arp request for the upstream router mac | 11:17 |
| *** zhiwei has joined #openstack-kolla | 11:17 | |
| sean-k-mooney | does your tor require the traffic to be vlan tagged with a specific vlan to reach the upstream rourter maybe? | 11:17 |
| tyrola | Mh ok. I need to say I am completely new to network namespaces and neutron itself. | 11:18 |
| *** Jeffrey4l_ has quit IRC | 11:18 | |
| tyrola | No vlan tags needed, the vlan's are managed by the router. | 11:19 |
| sean-k-mooney | am just as a quick test to double check can you try pinging the upstream router and confirm the packets are leaveing the host with tcpdump? | 11:20 |
| tyrola | http://paste.openstack.org/show/foxBRwJe2fIAYpMwYCML/ arp request seems to work | 11:20 |
| sean-k-mooney | cool | 11:20 |
| tyrola | tcpdump was running outside of the docker container | 11:20 |
| sean-k-mooney | so it looks like you are not getting an arp reply | 11:20 |
| *** zhiwei has quit IRC | 11:21 | |
| sean-k-mooney | am out of interest were you able to reach the router successfully before you deployed kolla? | 11:21 |
| tyrola | I am able to reach the router everywhere just not inside of the router namespace | 11:22 |
| tyrola | http://paste.openstack.org/show/lAqCJhIIr3I4di2eEtMz/ as you can see it just doesn't work if I am inside the qrouter namespace | 11:22 |
| sean-k-mooney | out side of the router namespace you would be using a different network interface though | 11:23 |
| sean-k-mooney | i would guess your are reaching it via the internal interface | 11:24 |
| tyrola | [root@neutron02 ~]# ping 5.83.160.1 -I external | 11:24 |
| tyrola | ping: Warning: source address might be selected on device other than external. | 11:24 |
| tyrola | PING 5.83.160.1 (5.83.160.1) from 10.0.0.22 external: 56(84) bytes of data. | 11:24 |
| tyrola | From 10.0.0.22 icmp_seq=1 Destination Host Unreachable | 11:24 |
| tyrola | From 10.0.0.22 icmp_seq=2 Destination Host Unreachable | 11:24 |
| tyrola | It seems you are right :( | 11:24 |
| sean-k-mooney | im assuming this is a production setup. if not the simple answer is to nat the traffic | 11:25 |
| tyrola | Yes, this should be a production env soon | 11:26 |
| openstackgerrit | Paul Bourke proposed openstack/kolla: Document a common Ceph bootstrap failure scenario https://review.openstack.org/334422 | 11:29 |
| *** mewald has joined #openstack-kolla | 11:30 | |
| *** Jeffrey4l_ has joined #openstack-kolla | 11:30 | |
| tyrola | Yes, this should be a production env soon | 11:32 |
| *** Mr_Broken has quit IRC | 11:36 | |
| *** Mr_Broken has joined #openstack-kolla | 11:37 | |
| *** mliima has joined #openstack-kolla | 11:38 | |
| *** Jeffrey4l_ has quit IRC | 11:38 | |
| *** neilus1 has joined #openstack-kolla | 11:39 | |
| tyrola | sean-k-mooney: Seems like the switch ports aren't in the correct vlan... we will fix it asap I think thats the issue | 11:39 |
| tyrola | Thank you very much for helping me. It not fixed our issue but I've learned how to debug neutron a little bit more :) | 11:39 |
| *** neilus has quit IRC | 11:40 | |
| *** neilus1 has quit IRC | 11:47 | |
| openstackgerrit | Merged openstack/kolla: Add unit test for stop and restart container https://review.openstack.org/331524 | 11:51 |
| *** Mr_Broken has quit IRC | 11:51 | |
| *** Mr_Broken has joined #openstack-kolla | 11:54 | |
| sean-k-mooney | tyrola: glad i could help. hopefully you will have it working soon | 11:55 |
| *** haplo37 has quit IRC | 11:55 | |
| mandre | pbourke: not sure my -1 for workflow is going to change anything in https://review.openstack.org/#/c/334237/ | 11:57 |
| patchbot | mandre: patch 334237 - kolla (stable/liberty) - Create ansible home directory in kolla-toolbox con... | 11:57 |
| *** Jeffrey4l_ has joined #openstack-kolla | 11:58 | |
| pbourke | should do | 11:58 |
| pbourke | removed mine too | 11:58 |
| *** rhallisey has joined #openstack-kolla | 12:00 | |
| *** neilus has joined #openstack-kolla | 12:04 | |
| *** haplo37 has joined #openstack-kolla | 12:04 | |
| *** diga has quit IRC | 12:11 | |
| *** williamcaban has joined #openstack-kolla | 12:14 | |
| bootsha | hi Martin and Paul, thanks for the review of https://review.openstack.org/#/c/334237/. Sorry for my ignorance, but As suggested by Martin, Should I remove the exiting patch set completely and submit a fresh one with git cherry-pick -x ? | 12:15 |
| patchbot | bootsha: patch 334237 - kolla (stable/liberty) - Create ansible home directory in kolla-toolbox con... | 12:15 |
| *** ppowell_ has joined #openstack-kolla | 12:23 | |
| openstackgerrit | Merged openstack/kolla: Document a common Ceph bootstrap failure scenario https://review.openstack.org/334422 | 12:28 |
| rhallisey | bootsha, sure that works | 12:28 |
| mandre | bootsha: you don't have to abandon your current patch if you modify the commit message to add a line containing (cherry-picked from commit ...) | 12:29 |
| *** daneyon has joined #openstack-kolla | 12:30 | |
| bootsha | got it thanks. I will probably add cherry-picked from commit ... | 12:30 |
| mandre | bootsha: in your case, you need to add (cherry-pick from commit 76f97b406735540eb86fb3f293343ed4803c45d7) | 12:30 |
| mandre | bootsha: fyi, that's the only thing that 'git cherry-pick -x' does for you, it adds this line to the commit message | 12:31 |
| bootsha | yea, I saw that, but was in doubt :) | 12:31 |
| *** daneyon has quit IRC | 12:34 | |
| openstackgerrit | Mohammed Salih Puthenpurayil proposed openstack/kolla: Create ansible home directory in kolla-toolbox container https://review.openstack.org/334237 | 12:39 |
| openstackgerrit | Mohammed Salih Puthenpurayil proposed openstack/kolla: Create ansible home directory in kolla-toolbox container https://review.openstack.org/334239 | 12:40 |
| *** bootsha has quit IRC | 12:41 | |
| *** zhiwei has joined #openstack-kolla | 12:42 | |
| *** haplo37 has quit IRC | 12:44 | |
| *** salv-orlando has joined #openstack-kolla | 12:44 | |
| *** ccesario has joined #openstack-kolla | 12:45 | |
| *** g3ek has quit IRC | 12:45 | |
| *** haplo37 has joined #openstack-kolla | 12:46 | |
| *** g3ek has joined #openstack-kolla | 12:46 | |
| *** zhiwei has quit IRC | 12:47 | |
| *** salv-orlando has quit IRC | 12:51 | |
| *** matrohon has joined #openstack-kolla | 12:52 | |
| *** williamcaban has quit IRC | 12:55 | |
| tyrola | sean-k-mooney: works now fine :) thanks again | 12:55 |
| *** williamcaban has joined #openstack-kolla | 12:57 | |
| sean-k-mooney | no worries. i hit a similar issue before glad i could help | 13:00 |
| *** coolsvap has quit IRC | 13:00 | |
| *** klint has quit IRC | 13:01 | |
| *** williamcaban has quit IRC | 13:01 | |
| *** Mr_Broke_ has joined #openstack-kolla | 13:07 | |
| *** Mr_Brok__ has joined #openstack-kolla | 13:09 | |
| *** sdake has joined #openstack-kolla | 13:10 | |
| *** Mr_Broken has quit IRC | 13:10 | |
| *** Mr_Broke_ has quit IRC | 13:11 | |
| *** jtriley has joined #openstack-kolla | 13:14 | |
| *** williamcaban has joined #openstack-kolla | 13:16 | |
| *** zhurong has joined #openstack-kolla | 13:18 | |
| *** sdake_ has joined #openstack-kolla | 13:20 | |
| *** sdake has quit IRC | 13:21 | |
| *** Mr_Brok__ has quit IRC | 13:27 | |
| *** Mr_Broken has joined #openstack-kolla | 13:27 | |
| *** inc0 has joined #openstack-kolla | 13:28 | |
| *** diogogmt has quit IRC | 13:29 | |
| *** haplo37 has quit IRC | 13:29 | |
| *** g3ek has quit IRC | 13:30 | |
| *** belmoreira has quit IRC | 13:31 | |
| *** Mr_Broke_ has joined #openstack-kolla | 13:34 | |
| *** haplo37 has joined #openstack-kolla | 13:35 | |
| *** g3ek has joined #openstack-kolla | 13:35 | |
| *** Mr_Broken has quit IRC | 13:36 | |
| *** Mr_Broken has joined #openstack-kolla | 13:38 | |
| *** Mr_Brok__ has joined #openstack-kolla | 13:40 | |
| *** Mr_Broke_ has quit IRC | 13:40 | |
| *** banix has joined #openstack-kolla | 13:41 | |
| *** Mr_Broken has quit IRC | 13:42 | |
| *** inc0 has quit IRC | 13:44 | |
| *** inc0 has joined #openstack-kolla | 13:44 | |
| *** aernhart has joined #openstack-kolla | 13:45 | |
| *** kangh_ has joined #openstack-kolla | 13:46 | |
| sdake_ | morning | 13:47 |
| kangh_ | hey sdake | 13:47 |
| *** salv-orlando has joined #openstack-kolla | 13:48 | |
| inc0 | howdy folks | 13:49 |
| mliima | morning all | 13:49 |
| *** Mr_Broken has joined #openstack-kolla | 13:52 | |
| *** b_bezak has quit IRC | 13:52 | |
| *** Mr_Brok__ has quit IRC | 13:55 | |
| *** prithiv has quit IRC | 13:55 | |
| *** Mr_Broke_ has joined #openstack-kolla | 13:55 | |
| *** kproskurin has joined #openstack-kolla | 13:55 | |
| *** salv-orlando has quit IRC | 13:55 | |
| *** kangh_ has left #openstack-kolla | 13:56 | |
| *** bootsha has joined #openstack-kolla | 13:56 | |
| *** Mr_Broken has quit IRC | 13:57 | |
| openstackgerrit | Mohammed Salih Puthenpurayil proposed openstack/kolla: Trivial fixes to correct couple of typos. https://review.openstack.org/334306 | 13:58 |
| *** huikang has joined #openstack-kolla | 13:59 | |
| *** Mr_Broke_ has quit IRC | 14:04 | |
| *** Mr_Broken has joined #openstack-kolla | 14:04 | |
| *** wmiller has quit IRC | 14:05 | |
| *** wmiller has joined #openstack-kolla | 14:05 | |
| *** Mr_Broke_ has joined #openstack-kolla | 14:09 | |
| *** Mr_Broken has quit IRC | 14:11 | |
| *** diogogmt has joined #openstack-kolla | 14:12 | |
| *** bootsha has quit IRC | 14:13 | |
| *** zhurong has quit IRC | 14:13 | |
| *** b_bezak has joined #openstack-kolla | 14:15 | |
| *** haplo37 has quit IRC | 14:16 | |
| *** g3ek has quit IRC | 14:17 | |
| *** g3ek has joined #openstack-kolla | 14:17 | |
| *** daneyon has joined #openstack-kolla | 14:18 | |
| *** haplo37 has joined #openstack-kolla | 14:18 | |
| *** Mr_Broken has joined #openstack-kolla | 14:20 | |
| *** ayoung has joined #openstack-kolla | 14:22 | |
| *** Mr_Brok__ has joined #openstack-kolla | 14:22 | |
| *** Mr_Broke_ has quit IRC | 14:22 | |
| *** daneyon has quit IRC | 14:23 | |
| *** Mr_Broken has quit IRC | 14:25 | |
| *** ssurana has joined #openstack-kolla | 14:27 | |
| *** salv-orlando has joined #openstack-kolla | 14:30 | |
| *** Mr_Brok__ has quit IRC | 14:36 | |
| *** Mr_Broken has joined #openstack-kolla | 14:43 | |
| *** zhiwei has joined #openstack-kolla | 14:44 | |
| *** ayoung_ has joined #openstack-kolla | 14:48 | |
| *** zhiwei has quit IRC | 14:48 | |
| *** JeroenBo has quit IRC | 14:49 | |
| *** belmoreira has joined #openstack-kolla | 14:50 | |
| *** david-lyle_ has joined #openstack-kolla | 14:51 | |
| openstackgerrit | Steven Dake proposed openstack/kolla: Test of mitaka gate https://review.openstack.org/334519 | 14:55 |
| *** david-lyle has quit IRC | 14:55 | |
| sdake_ | does anyone have any idea why this gat efialure is occuring | 14:56 |
| sdake_ | http://logs.openstack.org/10/333110/1/check/gate-kolla-dsvm-deploy-centos-binary/a2673af/console.html.gz#_2016-06-23_05_32_19_567362 | 14:56 |
| sdake_ | it happens all the time | 14:57 |
| *** sacharya has joined #openstack-kolla | 14:57 | |
| *** sdake has joined #openstack-kolla | 15:01 | |
| *** mliima has quit IRC | 15:01 | |
| *** mewald has quit IRC | 15:01 | |
| *** mewald has joined #openstack-kolla | 15:02 | |
| *** aernhart has quit IRC | 15:02 | |
| sdake | inc0 extradb | 15:03 |
| sdake | i want to talk about it | 15:03 |
| sdake | xtradb | 15:03 |
| sdake | or whatever it is | 15:03 |
| inc0 | you mean percona stuff? | 15:03 |
| sdake | right | 15:03 |
| sdake | do we need it? | 15:03 |
| *** sdake_ has quit IRC | 15:03 | |
| *** aernhart has joined #openstack-kolla | 15:03 | |
| inc0 | well, it does provide some cool features | 15:03 |
| sdake | another way to word that is | 15:04 |
| inc0 | I don't have much experience with it | 15:04 |
| sdake | can we operate without it | 15:04 |
| inc0 | we operate without it now | 15:04 |
| sdake | xtradb is definately used | 15:04 |
| inc0 | what percona gives you is for example simple backups | 15:04 |
| sdake | cool so that wfm but i want to operate without it | 15:04 |
| inc0 | ok, it is there | 15:07 |
| sdake | i dont even know what it does | 15:07 |
| inc0 | it's a toolset for mariadb | 15:07 |
| inc0 | that helps you with certain tasks | 15:07 |
| sdake | when i was debugging it about 6 months ago because it was busted | 15:07 |
| inc0 | not required at all afaik | 15:07 |
| sdake | i noticed only one part of percona is used | 15:09 |
| inc0 | xtrabackup right | 15:09 |
| *** mewald has quit IRC | 15:09 | |
| inc0 | ? | 15:09 |
| sdake | yup | 15:09 |
| sdake | xtradbbackup i think | 15:09 |
| inc0 | yeah, this helps with bit-level backup | 15:09 |
| sdake | kolla doesn't work at all without it | 15:09 |
| inc0 | not sure if galera uses this | 15:09 |
| inc0 | it might actually | 15:09 |
| sdake | i know other peopel have gotten replication with galera to work without it | 15:09 |
| inc0 | soo...I'd lie if I'd say that this is my area of expertise | 15:09 |
| *** david-lyle_ is now known as david-lyle | 15:09 | |
| sdake | did you add it? | 15:09 |
| inc0 | no. Sam did | 15:09 |
| sdake | i see | 15:09 |
| inc0 | so I don't think it needs | 15:09 |
| sdake | trying to come up for air here, any chance anyone can look into removing it | 15:09 |
| inc0 | percona | 15:09 |
| sdake | or making it optional? | 15:09 |
| *** prithiv has joined #openstack-kolla | 15:09 | |
| inc0 | galera doesn't seem to need percona | 15:09 |
| sdake | kolla crashes without xtradb | 15:09 |
| inc0 | but how crashes? what's the error? | 15:09 |
| sdake | i dont recall it was 6 months ago i looked at it | 15:09 |
| sean-k-mooney | sdake: it should be able to run with InnoDB | 15:09 |
| sdake | sean-k-mooney morning fine sir | 15:09 |
| sdake | sean-k-mooney so my diagram was a talking point | 15:09 |
| sean-k-mooney | xtradb is a perfromace optimsed version of InnoDB | 15:09 |
| *** ayoung has quit IRC | 15:09 | |
| inc0 | https://github.com/openstack/kolla/blob/6e8f01ca6a02f77a281df78f544cace063c78511/ansible/roles/mariadb/templates/wsrep-notify.sh.j2#L60 sdake | 15:09 |
| sean-k-mooney | sdake: and yes morning | 15:09 |
| *** ayoung_ is now known as ayoung | 15:09 | |
| sdake | sean-k-mooney if you think it should be done a different way please let me knwo so i can modify the digram | 15:09 |
| inc0 | I think if you change config of galera to not use xtrabackup, it might work | 15:10 |
| *** mliima has joined #openstack-kolla | 15:10 | |
| *** belmoreira has quit IRC | 15:10 | |
| *** tyrola has quit IRC | 15:11 | |
| sdake | any idea how to do that | 15:11 |
| sdake | I dont know jack shit about galera | 15:11 |
| *** tyrola has joined #openstack-kolla | 15:11 | |
| wirehead_ | kfox1111_: yeah, I know we need to address the issue of PersistentVolumeClaims but haven't worried about it yet. | 15:12 |
| sdake | wtb new redidck movie | 15:12 |
| sdake | morning wirehead_ | 15:12 |
| wirehead_ | kfox1111_: If we were to be starting over from zero without the existing working Kolla codebase, I'd totally try to build Kolla-kubernetes around Helm instead of Ansible. | 15:13 |
| wirehead_ | gmorning | 15:16 |
| sean-k-mooney | sdake: i have started working on rebasing the bifrost work today. | 15:16 |
| *** fragatina has quit IRC | 15:17 | |
| *** fragatina has joined #openstack-kolla | 15:17 | |
| sean-k-mooney | sdake: just googleing around i found this https://mariadb.com/kb/en/mariadb/mariadb-galera-cluster-known-limitations/ | 15:18 |
| sean-k-mooney | sdake: first bullet point "Currently replication works only with the InnoDB storage engine. " | 15:18 |
| sean-k-mooney | so you will need to have innoDB or xtraDB to use galara | 15:18 |
| *** ssurana has quit IRC | 15:21 | |
| *** Mr_Broken has quit IRC | 15:22 | |
| *** Mr_Broken has joined #openstack-kolla | 15:22 | |
| openstackgerrit | Merged openstack/kolla: Trivial fixes to correct couple of typos. https://review.openstack.org/334306 | 15:22 |
| *** coolsvap has joined #openstack-kolla | 15:24 | |
| Jeffrey4l_ | sdake, morning | 15:27 |
| openstackgerrit | Merged openstack/kolla: Add the verification of required images step before the deployment. https://review.openstack.org/333666 | 15:27 |
| Jeffrey4l_ | I create a test patch to build the image with ansible. https://review.openstack.org/334208 could u give us some idea for this? | 15:27 |
| coolsvap | Jeffrey4l_, hi there is no such rule as such | 15:28 |
| coolsvap | but yes i would like to have more eyes on it for sure | 15:29 |
| Jeffrey4l_ | coolsvap, thanks. | 15:29 |
| *** matrohon has quit IRC | 15:29 | |
| Jeffrey4l_ | the root cause i want to use ansible is: we need defend program in the dockerfile. | 15:30 |
| *** dmk0202 has quit IRC | 15:30 | |
| Jeffrey4l_ | Recently, the ceph packaging changed and it create ceph user now, which crashed the kolla. | 15:30 |
| Jeffrey4l_ | if using ansible, the solution is easy, just: `user: name=ceph`. | 15:31 |
| *** harlowja_at_home has joined #openstack-kolla | 15:31 | |
| *** daneyon has joined #openstack-kolla | 15:31 | |
| Jeffrey4l_ | if using raw dockerfile/RUN, we need `RUN id -u ceph &>/dev/null || useradd --user-group ceph`. | 15:31 |
| Jeffrey4l_ | which is ugly. | 15:31 |
| coolsvap | Jeffrey4l_, personally i like the approach it is very much in the lines for the elemental discussion we had | 15:32 |
| Jeffrey4l_ | the second reason is, ansible is more powerful. | 15:32 |
| ccesario | hey guys... due the needed of a cinder SVC driver, I need install multipathd into compute-node container (using the inc0 customize path) .... but I need that this service starts automatic, does someone know what is the best way of to do it ? | 15:32 |
| Jeffrey4l_ | yes. it will resolve the customize issue when building image. | 15:33 |
| *** daneyon_ has quit IRC | 15:34 | |
| *** Serlex has quit IRC | 15:35 | |
| *** ssurana has joined #openstack-kolla | 15:37 | |
| *** matrohon has joined #openstack-kolla | 15:38 | |
| openstackgerrit | Dipa Thakkar proposed openstack/kolla: Remove MariaDB warning from HAproxy https://review.openstack.org/334054 | 15:38 |
| *** Mr_Broken has quit IRC | 15:39 | |
| sdake | Jeffrey4l_ yo | 15:40 |
| Jeffrey4l_ | sdake, could you check above words i said? | 15:41 |
| sdake | Jeffrey4l_ i just read scrollback | 15:41 |
| sean-k-mooney | Jeffrey4l_: why do that check instead of using the ansible user module | 15:41 |
| sdake | i have your review up | 15:41 |
| sdake | looks daunting :) | 15:41 |
| Jeffrey4l_ | thanks. | 15:42 |
| sean-k-mooney | Jeffrey4l_: never mind missread RUN id -u ceph &>/dev/null || useradd --user-group ceph is in the docker file not ansible | 15:42 |
| *** pbourke has quit IRC | 15:43 | |
| *** pbourke has joined #openstack-kolla | 15:43 | |
| Jeffrey4l_ | sean-k-mooney, :) | 15:43 |
| *** jmccarthy has joined #openstack-kolla | 15:44 | |
| sean-k-mooney | ccesario if you want a standalone server to run on the compute node you should create a container and add it to the compute group | 15:44 |
| sean-k-mooney | s/server/service/ | 15:45 |
| *** haplo37 has quit IRC | 15:45 | |
| *** ssurana has quit IRC | 15:51 | |
| *** g3ek has quit IRC | 15:52 | |
| *** prithiv has quit IRC | 15:52 | |
| *** g3ek has joined #openstack-kolla | 15:54 | |
| sdake | it sure would be handy if pbr was someway used to generate tag ids for stable branches | 15:54 |
| sdake | Jeffrey4l_ had a look at the build work you did or ansible | 15:54 |
| sdake | i'm curious, what is the purposoe | 15:54 |
| sdake | orfor | 15:54 |
| sdake | is this a third alternative to the dsl and the customization via docker.j2? | 15:54 |
| *** Mr_Broken has joined #openstack-kolla | 15:54 | |
| Jeffrey4l_ | 1. Defensive programming: Recently, the ceph packaging changed and it create ceph user now, which crashed the kolla. | 15:54 |
| Jeffrey4l_ | if using ansible, the solution is easy, just: `user: name=ceph`. | 15:54 |
| Jeffrey4l_ | if using raw dockerfile/RUN, we need `RUN id -u ceph &>/dev/null || useradd --user-group ceph`. | 15:54 |
| Jeffrey4l_ | there are two much such potential issue exist. | 15:55 |
| sdake | so your proposing rewriting all 130 dockerfiles as ansible? | 15:55 |
| *** haplo37 has joined #openstack-kolla | 15:55 | |
| Jeffrey4l_ | 2. it will resolve the customize issue when building image. and yes it just like a kind of DSL language. | 15:56 |
| *** b_bezak has quit IRC | 15:56 | |
| sdake | what about build performance | 15:56 |
| Jeffrey4l_ | yes. if the base is acceptable. I think we'd better change all the dockfiles. | 15:56 |
| sdake | we have a bunch of cli options to build.py | 15:57 |
| Jeffrey4l_ | sdake, the performance should almost the same. ansible do not add much logical. | 15:57 |
| sdake | how to implement those? | 15:57 |
| sdake | Jeffrey4l_ our build is multitheaded | 15:57 |
| sdake | ansible not right? | 15:57 |
| Jeffrey4l_ | sdake, the ansible is run in one container. so we can build multi container in one time. | 15:57 |
| sdake | my biggest concern is loss of all of the CLI options to build.py | 15:58 |
| sdake | inc0 ^^ | 15:58 |
| Jeffrey4l_ | most of the option is use by the build.py script. there is few is used by the dockerfile. | 15:59 |
| Jeffrey4l_ | see this https://github.com/openstack/kolla/blob/master/kolla/image/build.py#L626,L638 | 15:59 |
| Jeffrey4l_ | now, there only 14 value is used in the dockerfile. | 15:59 |
| Jeffrey4l_ | and we can pass the to the ansible-play by using `-e aaa=bbb` format. | 16:00 |
| Jeffrey4l_ | ansible-playbook | 16:00 |
| * inc0 reading log | 16:00 | |
| inc0 | Jeffrey4l_, you know our customization mechanism right? | 16:00 |
| Jeffrey4l_ | inc0, yes. | 16:01 |
| inc0 | so when ceph updated this user thign | 16:01 |
| inc0 | we could add this into our Dockerfiles | 16:01 |
| inc0 | and that's it right? | 16:01 |
| Jeffrey4l_ | normally, kolla should handle this. | 16:02 |
| Jeffrey4l_ | rather than fix by us. | 16:02 |
| inc0 | Jeffrey4l_, please repeat the issue, I don't really get it | 16:02 |
| Jeffrey4l_ | Defensive programming is important | 16:02 |
| Jeffrey4l_ | inc0, https://review.openstack.org/334168 | 16:02 |
| Jeffrey4l_ | here is the fix. | 16:02 |
| inc0 | what does defensive programming mean? | 16:03 |
| Jeffrey4l_ | inc0, https://en.wikipedia.org/wiki/Defensive_programming | 16:03 |
| Jeffrey4l_ | inc0, ceph issue http://logs.openstack.org/58/334158/1/check/gate-kolla-dsvm-build-ubuntu-source/c28e596/console.html#_2016-06-25_10_18_07_949815 | 16:05 |
| *** mewald has joined #openstack-kolla | 16:05 | |
| inc0 | ok I see | 16:06 |
| *** daneyon_ has joined #openstack-kolla | 16:06 | |
| inc0 | how is it possible that ceph user exists on newly built container? | 16:07 |
| *** Mr_Broken has quit IRC | 16:07 | |
| Jeffrey4l_ | i think someone/canical changed the ceph packages. and when installing ceph, the package will create the ceph user automatically. | 16:08 |
| *** Mr_Broken has joined #openstack-kolla | 16:08 | |
| *** vhosakot has joined #openstack-kolla | 16:09 | |
| *** sacharya has quit IRC | 16:09 | |
| Jeffrey4l_ | there is another approach. | 16:09 |
| Jeffrey4l_ | just like: | 16:09 |
| Jeffrey4l_ | 1. run a container with `sleep infinity` | 16:09 |
| Jeffrey4l_ | 2. using ansible + docker connection to initial the containers. ( install package and prepare the container env, just like it is a LXC) | 16:10 |
| sdake | http://paste.fedoraproject.org/385565/46704382/ -> http://paste.fedoraproject.org/385565/46704382 | 16:10 |
| sdake | liberty is broked^ | 16:10 |
| Jeffrey4l_ | 3. commit the container to a tag. | 16:10 |
| sean-k-mooney | Jeffrey4l_: what benifit does that have over a docker build? | 16:11 |
| *** daneyon_ has quit IRC | 16:11 | |
| Jeffrey4l_ | in this way, there is not Dockerfile anymore. there should be only one ansible playbooks, which will generate the kolla images. | 16:11 |
| Jeffrey4l_ | And we can reuse/extend the playbooks in different image. | 16:12 |
| inc0 | Jeffrey4l_, I'm -2 to this, we need containers to be ansible-agnosticv | 16:12 |
| *** sacharya has joined #openstack-kolla | 16:12 | |
| sdake | well you can't really be -2 to a spec | 16:12 |
| sdake | but i dont see a strong value or this work | 16:12 |
| inc0 | we already have services consuming just containers | 16:12 |
| sean-k-mooney | sdake: the only value i see with that approch would be if we wanted to support a different runtime e.g. rocket or appc | 16:13 |
| Jeffrey4l_ | sean-k-mooney, scrollback..I told serveral benefit. | 16:13 |
| sdake | Jeffrey4l_ what i'm getting at is the epain doesnt seem worth the gain | 16:13 |
| sdake | we need customizations | 16:14 |
| Jeffrey4l_ | sdake, i will push a follow patch to describle how the customization will looks like . | 16:14 |
| sdake | i am not super keen to rework the entire dockerfile syntax into something completely different that isn't coherent for what peopel are trained for 3 months before release:) | 16:14 |
| *** jmccarthy has left #openstack-kolla | 16:14 | |
| sdake | cn someon look at that keystone log i pasted pls | 16:15 |
| sdake | tip of stable/liberty | 16:15 |
| sdake | just built 15 minutes ago or so | 16:15 |
| Jeffrey4l_ | sdake, seem the keystone db is not synced. | 16:15 |
| *** ayoung has quit IRC | 16:16 | |
| sdake | ya it looked like bootstrap didnt go to me too | 16:16 |
| sdake | trying a redeploy | 16:16 |
| Jeffrey4l_ | sdake, need remove the keystone database. Or the whole db | 16:16 |
| sdake | i did a total cleanup | 16:17 |
| sdake | this is a fresh bulid | 16:17 |
| sdake | TASK: [keystone | Running Keystone bootstrap container] *********************** | 16:18 |
| sdake | skipping: [localhost] | 16:18 |
| sdake | TASK: [keystone | Starting keystone container] ******************************** | 16:18 |
| sdake | changed: [localhost] | 16:18 |
| sdake | TASK: [keystone | Wait for keystone startup] ********************************** | 16:18 |
| sdake | ok: [localhost] | 16:18 |
| sdake | sorry for spam | 16:18 |
| sdake | why would it skip bootstrap? | 16:18 |
| Jeffrey4l_ | skipping.. | 16:18 |
| Jeffrey4l_ | copy the above log out. | 16:20 |
| Jeffrey4l_ | sdake, ^ | 16:20 |
| sdake | huh? | 16:20 |
| sdake | you mean paste entire output? | 16:20 |
| Jeffrey4l_ | right. the logs before Running Keystone bootstrap container | 16:21 |
| Jeffrey4l_ | just 10-20 log should be enough. | 16:21 |
| sdake | ansible 1.9.4 on tip of stable/liberty | 16:22 |
| sdake | http://paste.fedoraproject.org/385574/4454514/ -> http://paste.fedoraproject.org/385574/04454514 | 16:22 |
| *** Mr_Broken has quit IRC | 16:22 | |
| sdake | seems to be skipping lots of things it should not | 16:23 |
| Jeffrey4l_ | sdake, i do not think you db is cleaned. | 16:24 |
| Jeffrey4l_ | TASK: [keystone | Creating Keystone database] ********************************* | 16:24 |
| Jeffrey4l_ | [0;32mok: [localhost -> localhost][0m | 16:24 |
| sdake | cleanup-containers - cleanup-host - cleanup-images | 16:24 |
| Jeffrey4l_ | this taks is OK rather than change. means you keystone database exist. | 16:24 |
| *** Mr_Broken has joined #openstack-kolla | 16:25 | |
| *** coolsvap has quit IRC | 16:26 | |
| sdake | there is no database on my system | 16:27 |
| inc0 | sdake, are volumes deleted? | 16:28 |
| sdake | let me check moment | 16:29 |
| *** coolsvap has joined #openstack-kolla | 16:29 | |
| inc0 | maybe you have stale volumes and that makes keystone bootstrap skip | 16:29 |
| sdake | http://paste.fedoraproject.org/385578/46704497/ -> http://paste.fedoraproject.org/385578/46704497 | 16:29 |
| sean-k-mooney | inc0: the cleanup contaiers script is ment to delete the volumns too right | 16:29 |
| inc0 | yeah, but sometimes fail to do so | 16:30 |
| inc0 | we had some fixes in master | 16:30 |
| *** ayoung has joined #openstack-kolla | 16:31 | |
| *** g3ek has quit IRC | 16:34 | |
| *** haplo37 has quit IRC | 16:34 | |
| sdake | well i just wiped out /vaar/lib/docker | 16:35 |
| sdake | lets see if it workie now | 16:35 |
| sean-k-mooney | sdake: thats rather a blunt hammer | 16:36 |
| sdake | I'LL DROP THE HAMMER | 16:36 |
| *** coolsvap has quit IRC | 16:37 | |
| *** haplo37 has joined #openstack-kolla | 16:37 | |
| *** harlowja_at_home has quit IRC | 16:37 | |
| *** g3ek has joined #openstack-kolla | 16:37 | |
| *** tyrola has quit IRC | 16:37 | |
| openstackgerrit | Md Nadeem proposed openstack/kolla: Add test for remove container, get state and get env https://review.openstack.org/331556 | 16:38 |
| sdake | Jeffrey4l_ ar you tlakingbout merging our deploy and bulid steps? | 16:41 |
| *** athomas has quit IRC | 16:46 | |
| openstackgerrit | Merged openstack/kolla: Create ansible home directory in kolla-toolbox container https://review.openstack.org/334239 | 16:48 |
| sdake | this weekend i hit 230 in the world for diablo barbarian | 16:48 |
| *** matrohon has quit IRC | 16:49 | |
| sdake | cracking the top 100 is near impossible - those cats must play 12 hours a daay | 16:50 |
| sdake | i beat the timer by 0.117 seconds | 16:50 |
| *** aNupoisc has joined #openstack-kolla | 16:51 | |
| sdake | bootstrap skipped again - completely clean /var/lib/docker | 16:52 |
| *** ntpttr_ has quit IRC | 16:56 | |
| sdake | would someone else try the tip of liberty? | 16:59 |
| *** salv-orl_ has joined #openstack-kolla | 16:59 | |
| *** belmoreira has joined #openstack-kolla | 17:00 | |
| jogam | morning gents, sorry to interrupt, but are the latest kolla/ubuntu-source-* images on docker hub compiled on/for Ubuntu 14.04 LTS | 17:00 |
| sdake | jogan we are in the process of moving to the kolla namespace | 17:01 |
| sdake | jogam so i am not sure wha tthe status is there - coolsvap has been doing the workk | 17:01 |
| jogam | ADUM: are there any images (latest being 3.0.0) for 16.04 LTS or would you avoid ubuntu | 17:01 |
| sdake | i would avoid master for now | 17:02 |
| sdake | and use mitaka (2.0.1) | 17:02 |
| jogam | sdake: was looking at the kolla/* namespace on docker, was that the one you meant with the move destination? | 17:02 |
| *** salv-orlando has quit IRC | 17:02 | |
| sdake | jogam yup | 17:02 |
| *** belmoreira has quit IRC | 17:03 | |
| sean-k-mooney | sdake: dose mitaka currently deploy with the stable mitaka branch of ansible? i had to swap to master to get it to work the last time i tried to deploy mitaka images | 17:04 |
| sdake | it should | 17:04 |
| sdake | whether it does or not is unknown at this time | 17:04 |
| sdake | i am trying liberty atm and it seems broken | 17:04 |
| *** ayoung has quit IRC | 17:06 | |
| openstackgerrit | Merged openstack/kolla: Enable openvswitch container logs in host volumes https://review.openstack.org/334283 | 17:06 |
| jogam | sdake or dockerhub pushers: what would you recommend to use for an eval system setup with three nodes to convince management to use kolla for OS deploy? (2.0.1 from DH/kolla, compile from current master, 3.0.0 from DH/kolla)? | 17:10 |
| sdake | jogan do following | 17:11 |
| sdake | git clone http://github.com/openstack/kolla | 17:11 |
| *** zhiwei has joined #openstack-kolla | 17:11 | |
| sdake | git checkout 2.0.1 | 17:11 |
| sdake | tools/build.py --registery YOUR_REGISSTRY_SERVER:4000 --push | 17:11 |
| jogam | awesome, e.g. build 2.0.1 from latest git clone | 17:11 |
| sdake | in other owrds, compile 2.0.1 and push to a local registry | 17:11 |
| jogam | scrolling throught he list this morning I saw some issues with CEPH, after having failed with Cinder on 16.04 LTS, does CEPH work on the 2.0.1 branch on virtual env with dual virt. HD? | 17:13 |
| jogam | and 14.04 LTS? | 17:13 |
| sdake | it worked when we released it | 17:13 |
| sdake | dont now about 16.04 | 17:13 |
| sdake | we dont support 16.04 in stable branches | 17:13 |
| jogam | I learned that the hard way :( | 17:13 |
| sdake | it "should" work but if it does or not I dont know | 17:14 |
| jogam | BUT: the highest ´likelyness´ that it works is currently with 2.0.1 from git latest? | 17:14 |
| sdake | 2.0.1 is a tag | 17:15 |
| sdake | so at that point atleat 6 people manually testedd it | 17:15 |
| sdake | and it worked | 17:15 |
| sdake | but the packaging changes | 17:15 |
| sdake | so that breaks tags unfortunately | 17:15 |
| *** zhiwei has quit IRC | 17:15 | |
| sdake | jogam you don't want to deploy maste rbecause that deploys ALL OF OPENSTACK as master | 17:17 |
| sdake | jogam your use case (eval) is best served by a stable branch | 17:18 |
| sdake | jogam which deploys either liberty (1.1.1) or mitaka (2.0.1) | 17:18 |
| jogam | ok, slowly starting to get your GitHub setup, somewhat :) | 17:19 |
| *** ssurana has joined #openstack-kolla | 17:20 | |
| sdake | where is the ceph resolution that Jeffrey4l_ was talkingabout? | 17:21 |
| sdake | jogan 3.0.0 is not yet released | 17:21 |
| sdake | we dont push to the docker hub any unreleased software | 17:21 |
| *** ayoung has joined #openstack-kolla | 17:22 | |
| *** Mr_Broken has quit IRC | 17:22 | |
| jogam | sdake: already there --> https://hub.docker.com/r/kolla/ubuntu-source-heka/tags | 17:23 |
| *** Mr_Broken has joined #openstack-kolla | 17:23 | |
| sdake | jogam it is going to be deleted | 17:23 |
| sdake | coolsvap did wrong when he pushed the 3.0.0 images | 17:23 |
| *** g3ek has quit IRC | 17:23 | |
| *** jtriley has quit IRC | 17:23 | |
| *** haplo37 has quit IRC | 17:23 | |
| jogam | ok, no worries, I´ll shut up and just monkey around here and compile the latest master and mitaka branches and then see what happens | 17:24 |
| *** g3ek has joined #openstack-kolla | 17:25 | |
| *** harlowja has quit IRC | 17:25 | |
| sdake | feel free to ask questions :) | 17:25 |
| sdake | i wouldn't use master though | 17:25 |
| *** haplo37 has joined #openstack-kolla | 17:25 | |
| jogam | sdake: one more, for mitaka its still ansible < 2.0.0? | 17:25 |
| sdake | because openstack master is always in a state of brokeness | 17:25 |
| sdake | right mitaka and liberty are ansible 1.9.4 | 17:26 |
| jogam | oh sweet, so everyone who is using OS is keeping bravely a couple of releases back? | 17:26 |
| *** harlowja has joined #openstack-kolla | 17:33 | |
| *** Mr_Broken has quit IRC | 17:40 | |
| openstackgerrit | Steven Dake proposed openstack/kolla: Use proper images when deploying mitaka https://review.openstack.org/334588 | 17:43 |
| sdake | jogam i just confirmed liberty is working | 17:45 |
| sdake | git checkout stable/liberty | 17:45 |
| *** salv-orl_ has quit IRC | 17:47 | |
| *** fragatina has quit IRC | 17:49 | |
| *** Mr_Broken has joined #openstack-kolla | 17:53 | |
| *** daneyon_ has joined #openstack-kolla | 17:55 | |
| jogam | sdake: thx, will go for that then with ansible 1.9.4, anything else version-wise that I should NOT do :) | 17:55 |
| sdake | docker 1.10 or later | 17:55 |
| sdake | jogam also change the #openstack_version: whateveritis" in /etc/kolla/globals.yml to openstack_version: "1.1.2" | 17:56 |
| sdake | if you use git checkout 2.0.1 | 17:56 |
| sdake | it will use the correcct image and you wont need to do this step | 17:57 |
| jogam | huh? so when git checkout 2.0.1 tag I get the liberty release which is 1.1.2?? | 17:57 |
| sdake | noep you get a release that is 1.1.1 | 17:57 |
| sdake | but after that, pbr autoincrementst he versoin to 1.1.2 | 17:57 |
| sdake | stable/liberty != 1.1.1 | 17:58 |
| sdake | stable/liberty has critical and high bug fixes | 17:58 |
| sdake | we release z streams every 45 days | 17:58 |
| *** mewald has left #openstack-kolla | 17:58 | |
| jogam | sdake: so you meant stable/mitaka is working above? | 17:59 |
| *** daneyon_ has quit IRC | 17:59 | |
| sdake | stable/mitaka != 2.0.1 | 17:59 |
| sdake | stable/mitaka gets the same bugfixes as stable/liberty | 17:59 |
| sdake | they ar essnetially teh same codebase | 18:00 |
| jogam | OK, so what branch is the tag 2.0.1 on? | 18:00 |
| sdake | with modifications to load liberty versus mitaka | 18:00 |
| sdake | jogam i didnt tag it so i'm not ure, but i assume its on the stable/mitaka branch | 18:00 |
| sdake | tag = artifact produced | 18:00 |
| sdake | branch = work in progress leading up to a tag | 18:00 |
| Daviey | sdake: Sorry, are you talking about using Master kolla to deploy older OpenStack versions tan 3.0.0? | 18:01 |
| sdake | Daviey no | 18:01 |
| Daviey | OK | 18:01 |
| jogam | sweet: just to recap ==> Ansible 1.9.4, docker > 1.10, git clone + checkout 2.0.1 update globals.yml to release v. 1.1.2 and then pray :) | 18:01 |
| sdake | our model is one branch per release of openstack | 18:01 |
| sdake | shouldn't have to pray | 18:01 |
| sdake | just ping us on the channel | 18:01 |
| sdake | someone can help you out | 18:01 |
| sdake | if you git checkout 2.0.1 you don't need to change globals.yml | 18:02 |
| sdake | (atleast for the release version :) | 18:02 |
| *** salv-orlando has joined #openstack-kolla | 18:03 | |
| *** salv-orlando has quit IRC | 18:06 | |
| *** salv-orlando has joined #openstack-kolla | 18:06 | |
| *** haplo37 has quit IRC | 18:09 | |
| *** diogogmt has quit IRC | 18:09 | |
| *** diogogmt has joined #openstack-kolla | 18:10 | |
| *** g3ek has quit IRC | 18:10 | |
| *** haplo37 has joined #openstack-kolla | 18:10 | |
| *** g3ek has joined #openstack-kolla | 18:11 | |
| *** shardy is now known as shardy_afk | 18:17 | |
| *** Mr_Broken has quit IRC | 18:22 | |
| *** Mr_Broken has joined #openstack-kolla | 18:22 | |
| jogam | sdake: for an eval system on VMWare, should I do: echo 'options kvm_intel nested=1' >> /etc/modprobe.d/qemu-system-x86.conf as well? | 18:23 |
| sdake | hrm | 18:23 |
| sdake | no read the philosophy document | 18:23 |
| sdake | it explains how to use qemu virt | 18:23 |
| sdake | kolla may or may not work on vmwre, depends on if multicast is enabled or disabled | 18:24 |
| jogam | sdake: did read, but also reading others using kolla and that was one of the recommendations: http://www.jinkit.com/openstack-dockerized/ | 18:24 |
| sdake | that may work | 18:25 |
| sdake | haven't tried it myself | 18:25 |
| sdake | qemu is your safest bet | 18:25 |
| jogam | sdake: how do you gage kolla-working, I got it starting the CirrOS image + networking for 3.0.0 from latest on Ubuntu 16.04 LTS but no way for cinder to get convinced to work, e.g. no volumes | 18:26 |
| jogam | yup, kvm is not working, only qemu! | 18:26 |
| sdake | ya nessted kvm is dessigned to run inside a parent kvm | 18:26 |
| jogam | but the first time I did not test the above command | 18:26 |
| sdake | not inside vmware | 18:26 |
| *** aNupoisc has quit IRC | 18:26 | |
| sdake | jogam so you have kolla functional but not cinder | 18:27 |
| *** ayoung has quit IRC | 18:27 | |
| jogam | yup, was originally from kolla deploy on top of openstack | 18:27 |
| sdake | i.e. you ahve compute kit working | 18:27 |
| jogam | sdake: well kind of, now not so much anymore, I am just on my way to trash it | 18:27 |
| jogam | had: 1 of each compute, ctrl, nw, and storage but ran into HD limitations for images and with Cinder into disappearing volumes | 18:28 |
| jogam | when I re-configured to Kolla deploy to have the image-registry on the storage node then images > CirrOS would time-out uploading and the show existent, but after navigation a login was required and the image was gone... | 18:29 |
| jogam | sdake: not sure about definition of ´compute kit´ | 18:29 |
| sdake | folks - this is critical - upgrades are fubared - Jeffrey4l_ can you look into working around this sproblem https://bugs.launchpad.net/kolla/+bug/1596653 | 18:31 |
| openstack | Launchpad bug 1596653 in kolla "upgrade fails if nova vm is launched" [Critical,Confirmed] | 18:31 |
| *** aNupoisc has joined #openstack-kolla | 18:32 | |
| openstackgerrit | David Wang proposed openstack/kolla-kubernetes: Update kolla-kubernetes installation instructions https://review.openstack.org/334616 | 18:32 |
| *** Mr_Broken has quit IRC | 18:37 | |
| *** banix has quit IRC | 18:38 | |
| *** jtriley has joined #openstack-kolla | 18:40 | |
| *** salv-orlando has quit IRC | 18:40 | |
| *** Mr_Broken has joined #openstack-kolla | 18:44 | |
| *** b_bezak has joined #openstack-kolla | 18:44 | |
| inc0 | sdake, looks like something local to your system, can you totally clean up OS and retry? | 18:46 |
| sdake | i just did | 18:46 |
| sdake | same reult | 18:46 |
| sdake | i deleted qemu-vm processes and was able to ugprade successfully | 18:46 |
| sdake | vm/kvm | 18:46 |
| sdake | libvirt was gone after the upgrade | 18:47 |
| sdake | so docker is deleting it - so it can be recreating | 18:47 |
| sdake | created | 18:47 |
| sdake | i think the solutoin si to ignore the error and start it again if there was one | 18:47 |
| sdake | or just ignore the error | 18:47 |
| sdake | you said this works for you with aufs? | 18:48 |
| inc0 | I'll try in a minute, but it did work with aufs | 18:48 |
| sdake | did you have a vm launched? | 18:48 |
| inc0 | yeah | 18:48 |
| inc0 | but docker version changed in the process | 18:49 |
| inc0 | I'll try to do it in a moment | 18:49 |
| *** Mr_Broken has quit IRC | 18:52 | |
| *** Mr_Broken has joined #openstack-kolla | 18:53 | |
| *** g3ek has quit IRC | 18:56 | |
| *** haplo37 has quit IRC | 18:56 | |
| *** g3ek has joined #openstack-kolla | 18:56 | |
| *** haplo37 has joined #openstack-kolla | 18:57 | |
| openstackgerrit | Steven Dake proposed openstack/kolla: Use proper images when deploying mitaka https://review.openstack.org/334588 | 18:57 |
| *** jtriley has quit IRC | 18:58 | |
| *** b_bezak has quit IRC | 19:02 | |
| ccesario | hey guys.... does someone have any idea about this error ? http://paste.openstack.org/show/523644/ http://paste.openstack.org/show/523645/ I need that from nova-compute container the multipathd client connect on multipathd socket running on host ... but I 'm getting coonnection refused when I try it from nova-compute container. | 19:02 |
| sdake | if connection refused, must need some type of bindmount | 19:03 |
| jgriffith | sbezverk: ping | 19:04 |
| *** sacharya_ has joined #openstack-kolla | 19:04 | |
| ccesario | sdake, any tip ? | 19:04 |
| sdake | not the faintest idea | 19:05 |
| ccesario | hmmm | 19:06 |
| sdake | possibly something in /var/run? | 19:06 |
| *** jtriley has joined #openstack-kolla | 19:06 | |
| sdake | it is possible it is a system level socket | 19:06 |
| sdake | i.e. it does not use a file to do io | 19:07 |
| sdake | you can list active sockets with netstat | 19:07 |
| *** sacharya has quit IRC | 19:07 | |
| ccesario | sdake, the socket run on /run/ (on the host) | 19:08 |
| ccesario | sdake, I think that I 'm wrong. ... | 19:10 |
| ccesario | http://paste.openstack.org/show/523646/ | 19:10 |
| sdake | you see that @ in front | 19:10 |
| sdake | that is a system level socket | 19:11 |
| sdake | that means it starts with \0 | 19:11 |
| sdake | i am not sure how to access that from inside a docker container | 19:11 |
| sdake | possible multipathd can take a socket path? | 19:11 |
| sdake | i think the corerct term is abstract socket | 19:12 |
| sdake | been awhlie :) | 19:12 |
| *** salv-orlando has joined #openstack-kolla | 19:12 | |
| ccesario | sdake, maybe this ? http://paste.openstack.org/show/523647/ | 19:13 |
| sdake | possible that /var/run was my first suggetion :) | 19:14 |
| sdake | but comute iirc bindmounts /run laready | 19:14 |
| sdake | and /var/run symlinks to /run | 19:14 |
| sdake | or visa versa | 19:14 |
| ccesario | yes yes, from container I can list the multipathd socket file | 19:14 |
| ccesario | but strace show me connection refused | 19:15 |
| sdake | what is strace tyring to access - filename? | 19:15 |
| ccesario | connect(3, {sa_family=AF_LOCAL, sun_path="/var/run/multipathd.sock"}, 110) = -1 ECONNREFUSED (Connection refused) | 19:16 |
| sdake | not EPRM | 19:16 |
| sdake | EPERM | 19:16 |
| ccesario | http://paste.openstack.org/show/523644/ | 19:17 |
| *** salv-orlando has quit IRC | 19:17 | |
| sdake | can you ls /var/run/multipathd.sock from the container? | 19:17 |
| ccesario | yes | 19:17 |
| ccesario | http://paste.openstack.org/show/523645/ | 19:17 |
| *** berendt has joined #openstack-kolla | 19:20 | |
| sdake | ccesario is multipathd running on your host? | 19:20 |
| berendt | Can we please merge https://review.openstack.org/#/c/331430/ ? | 19:20 |
| patchbot | berendt: patch 331430 - kolla - Add reconfigure tasks for ceilometer ansible role | 19:20 |
| *** banix has joined #openstack-kolla | 19:20 | |
| ccesario | sdake, yes | 19:21 |
| sdake | inc0 that request is aimed at you ^^ :) | 19:21 |
| berendt | And can the core reviewers please check https://review.openstack.org/#/c/331573/, I do not understand why I review with the same change was merged 6 days after I opened this one... | 19:21 |
| patchbot | berendt: patch 331573 - kolla - Fix glance configuration templates (ABANDONED) | 19:21 |
| openstackgerrit | Ken Wronkiewicz proposed openstack/kolla-kubernetes: Adding debugging documentation for Keystone https://review.openstack.org/330788 | 19:21 |
| ccesario | sdake, http://paste.openstack.org/show/523648/ | 19:21 |
| sdake | berendt we have a large core review team different people merge different things | 19:22 |
| berendt | sdake do they not review open review requests first? | 19:22 |
| inc0 | berendt, sometimes we do;) however it happened to me too that I posted a patch where similar one was already in review | 19:23 |
| inc0 | shouldn't do it ofc | 19:23 |
| *** sacharya has joined #openstack-kolla | 19:24 | |
| *** Mr_Broken has quit IRC | 19:25 | |
| sean-k-mooney | ccesario: any messages from selinux/apparmor in dmesg? | 19:26 |
| berendt | inc0 just think that it is a waste of time to identify and fix issues two times. anyway, the issue is now fixed in the master branch :) | 19:26 |
| sean-k-mooney | ccesario: although maybe not as its not a permission denied | 19:26 |
| ccesario | sean-k-mooney, all of then disabled :) | 19:26 |
| *** banix has quit IRC | 19:27 | |
| openstackgerrit | Merged openstack/kolla: Add reconfigure tasks for ceilometer ansible role https://review.openstack.org/331430 | 19:27 |
| ccesario | sean-k-mooney, on host of course | 19:27 |
| *** salv-orlando has joined #openstack-kolla | 19:27 | |
| *** sacharya_ has quit IRC | 19:28 | |
| *** Mr_Broken has joined #openstack-kolla | 19:31 | |
| *** Jeffrey4l__ has joined #openstack-kolla | 19:32 | |
| *** Jeffrey4l_ has quit IRC | 19:34 | |
| *** harlowja has quit IRC | 19:36 | |
| *** haplo37 has quit IRC | 19:37 | |
| *** sdake has quit IRC | 19:38 | |
| *** sdake has joined #openstack-kolla | 19:40 | |
| sdake | berendt we have 8 pages of reviews open | 19:41 |
| sdake | berendt i think we are lucky to get through 3-4 merges a day | 19:41 |
| *** ayoung has joined #openstack-kolla | 19:42 | |
| sdake | inc0 so if i run upgrade and hat fails, and i run upgrade again it works | 19:43 |
| sdake | in the meantime libevirt was deleted | 19:44 |
| sdake | so i think a viable workaround is just to ignore the delete failure | 19:44 |
| sdake | and carry on | 19:44 |
| inc0 | if it works | 19:44 |
| inc0 | I'm building liberty now | 19:44 |
| *** haplo37 has joined #openstack-kolla | 19:46 | |
| openstackgerrit | Merged openstack/kolla: Use proper images when deploying mitaka https://review.openstack.org/334588 | 19:48 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Liberty is now on 1.1.2 prerelease https://review.openstack.org/334637 | 19:49 |
| *** b_bezak has joined #openstack-kolla | 19:51 | |
| *** harlowja has joined #openstack-kolla | 19:52 | |
| *** Mr_Broken has quit IRC | 19:52 | |
| *** Mr_Broken has joined #openstack-kolla | 19:53 | |
| sdake | inc0 i dont see where libvirt is stopped | 19:54 |
| sdake | in the playbooks | 19:54 |
| sdake | only that it is started | 19:54 |
| *** ppowell_ has quit IRC | 19:54 | |
| inc0 | sdake, kolla_docker redeploys container if it differs | 19:54 |
| inc0 | start.yml will redeploy container | 19:55 |
| *** banix has joined #openstack-kolla | 19:55 | |
| ccesario | sean-k-mooney, sdake suggestions about multipathd?! :) | 19:57 |
| *** b_bezak has quit IRC | 19:57 | |
| *** gfidente has quit IRC | 19:59 | |
| *** aNupoisc has quit IRC | 20:01 | |
| berendt | sdake Hope that I can help with more reviews in the next weeks. | 20:05 |
| sdake | berendt any help is appreciated :) | 20:06 |
| sdake | fwiw i htink most of our reviews have -1 on them for the most part | 20:07 |
| *** Mr_Broken has quit IRC | 20:07 | |
| *** Mr_Broken has joined #openstack-kolla | 20:09 | |
| sdake | inc0 this works http://paste.fedoraproject.org/385669/5824914/ -> http://paste.fedoraproject.org/385669/05824914 | 20:11 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Fix upgrades fail to upgrade https://review.openstack.org/334644 | 20:14 |
| *** alan_ has joined #openstack-kolla | 20:17 | |
| *** alan_ is now known as Guest52772 | 20:17 | |
| *** sdake_ has joined #openstack-kolla | 20:19 | |
| *** aernhart has quit IRC | 20:19 | |
| *** sdake has quit IRC | 20:22 | |
| inc0 | sdake_, that won't do good | 20:22 |
| inc0 | that will leave liberty libvirt | 20:22 |
| inc0 | and that's not great | 20:22 |
| sdake_ | nah libvirt gts deleted | 20:22 |
| sdake_ | tested manually first | 20:23 |
| inc0 | ehh | 20:23 |
| inc0 | no errors shoudl pass silently | 20:23 |
| inc0 | (import this) | 20:23 |
| *** Mr_Broken has quit IRC | 20:23 | |
| *** haplo37 has quit IRC | 20:23 | |
| *** g3ek has quit IRC | 20:23 | |
| sdake_ | huh | 20:23 |
| *** Mr_Broken has joined #openstack-kolla | 20:24 | |
| sdake_ | the remove operation actually removes th container | 20:25 |
| *** williamcaban has quit IRC | 20:25 | |
| sdake_ | however the remoe operation raises an exception in the process | 20:25 |
| sdake_ | this causes upgrade to crater | 20:25 |
| sdake_ | we dont want to have to run upgrade for every compute node twice | 20:25 |
| sdake_ | its not erally an error, more like a leak warning :) | 20:27 |
| *** aNupoisc has joined #openstack-kolla | 20:30 | |
| *** g3ek has joined #openstack-kolla | 20:32 | |
| *** haplo37 has joined #openstack-kolla | 20:33 | |
| *** jtriley has quit IRC | 20:35 | |
| *** Mr_Broken has quit IRC | 20:38 | |
| *** Mr_Broken has joined #openstack-kolla | 20:40 | |
| *** mliima has quit IRC | 20:43 | |
| *** sdake has joined #openstack-kolla | 20:49 | |
| openstackgerrit | Steven Dake proposed openstack/kolla: Liberty is now on 1.1.2 prerelease https://review.openstack.org/334637 | 20:51 |
| *** sdake_ has quit IRC | 20:53 | |
| openstackgerrit | Steven Dake proposed openstack/kolla: Update version info in config file https://review.openstack.org/334660 | 20:53 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Fix upgrades fail to upgrade https://review.openstack.org/334661 | 20:53 |
| *** berendt has quit IRC | 20:53 | |
| *** haplo37 has quit IRC | 21:08 | |
| openstackgerrit | Joshua Harlow proposed openstack/kolla: Be smarter about what to do when making a docker client https://review.openstack.org/330171 | 21:09 |
| *** Mr_Broken has quit IRC | 21:12 | |
| *** sdake_ has joined #openstack-kolla | 21:12 | |
| *** Guest52772 is now known as aernhart | 21:12 | |
| *** sdake has quit IRC | 21:13 | |
| jogam | sdake: nothing as stated works: when commands issued as described above (clone + checkout 2.0.1) an attempt is made to compile the latest (3.0.0 - according to docker reg) modules for centos, but I am using ubuntu! | 21:15 |
| *** sdake has joined #openstack-kolla | 21:16 | |
| jogam | can also not find any configuration for the ´tag´ conf/cfg parameter | 21:16 |
| sbezverk | jogam: version tag is defined in globals.yaml | 21:16 |
| *** haplo37 has joined #openstack-kolla | 21:17 | |
| sbezverk | # Valid option is Docker repository tag | 21:17 |
| sbezverk | openstack_release: "3.0.0" | 21:17 |
| *** sdake_ has quit IRC | 21:17 | |
| jogam | yup found it now, was using the /etc/kolla/globals.yml | 21:18 |
| harlowja | sdake inc0 got any time to explain the config stuff in kolla? | 21:20 |
| harlowja | how i work, what it does :-P | 21:20 |
| inc0 | harlowja, | 21:20 |
| inc0 | sooo | 21:20 |
| harlowja | soooo | 21:20 |
| *** Mr_Broken has joined #openstack-kolla | 21:20 | |
| harlowja | so | 21:20 |
| inc0 | let me get code up | 21:20 |
| *** rhallisey has quit IRC | 21:21 | |
| harlowja | i like the codes | 21:21 |
| sdake | harlowja you mean ini merging? | 21:21 |
| inc0 | https://github.com/openstack/kolla/blob/master/ansible/roles/mariadb/tasks/config.yml#L10 | 21:21 |
| harlowja | more the config.json stuff | 21:21 |
| inc0 | ahh | 21:21 |
| sdake | yup ive got time | 21:21 |
| sdake | moment let me find a link - lets start at the top | 21:21 |
| inc0 | config.json basically tells container which command will be called and which files should be copied from where to where | 21:21 |
| harlowja | like i'm using puppet and not ansible, guess i gotta have something similar to that | 21:21 |
| harlowja | inc0 k | 21:22 |
| inc0 | but basically how config in kolla works: | 21:22 |
| inc0 | task I linked will render config and put it into kolla_config_directoryu | 21:22 |
| sdake | harlowja https://github.com/openstack/kolla/blob/master/ansible/roles/heat/tasks/config.yml#L12 | 21:22 |
| inc0 | container then bindmount this directory to /var/lib/kolla | 21:22 |
| sdake | harlowja this is in every ansible playbook | 21:22 |
| harlowja | right | 21:23 |
| sdake | inc0 he wants to see code examples | 21:23 |
| harlowja | i'm ok with either :-P | 21:23 |
| inc0 | sdake, I linked it above... | 21:23 |
| harlowja | i take what i can get, not to picky, lol | 21:23 |
| inc0 | so container bindmount this | 21:23 |
| sdake | harlowja so ansible copies the config.json file for the container | 21:23 |
| inc0 | and look for config.json | 21:23 |
| inc0 | then checks files in config.json and copies them over to /etc | 21:23 |
| harlowja | right, that's via some script put into the container from kolla right? | 21:23 |
| inc0 | so all puppet has to do is to put these files there and start container with it bindmounted to /var/lib/docker | 21:24 |
| sdake | harlowja this is the json format; https://github.com/openstack/kolla/blob/master/ansible/roles/heat/templates/heat-api.json.j2 | 21:24 |
| inc0 | yeah, no ansible involved aside from placing these files in host dirt | 21:24 |
| inc0 | dir | 21:24 |
| harlowja | k | 21:24 |
| sdake | you can see it has the command to run and hee config files to copy around | 21:24 |
| inc0 | so it's actually pretty easy to do the same with puppet | 21:24 |
| harlowja | just format like https://github.com/openstack/kolla/blob/master/ansible/roles/heat/templates/heat-api.json.j2 has to be made | 21:24 |
| harlowja | *not a template | 21:25 |
| inc0 | yeah | 21:25 |
| harlowja | k | 21:25 |
| sdake | harlowja for whatever container you want pretty much | 21:25 |
| inc0 | well, at the end of the day it has to be json | 21:25 |
| sdake | harlowja that json is parsed by https://github.com/openstack/kolla/blob/master/docker/base/set_configs.py | 21:25 |
| *** aernhart has quit IRC | 21:26 | |
| harlowja | ya, i gotta look at that | 21:26 |
| *** aernhart has joined #openstack-kolla | 21:26 | |
| sdake | every container inherits this cmd https://github.com/openstack/kolla/blob/master/docker/base/Dockerfile.j2#L262 | 21:26 |
| harlowja | kk | 21:27 |
| sdake | so when we make a heat-api container eventually it ends up running "kolla_start" | 21:27 |
| harlowja | yup yup, i've seen that | 21:27 |
| sdake | this ends up running this script: | 21:27 |
| sdake | https://github.com/openstack/kolla/blob/master/docker/base/start.sh | 21:27 |
| *** aernhart has quit IRC | 21:27 | |
| sdake | so every container we make runs start.sh | 21:27 |
| sdake | same start.sh | 21:28 |
| sdake | via inheritence from base image | 21:28 |
| sdake | and inheritence of CMD operation | 21:28 |
| *** aernhart has joined #openstack-kolla | 21:28 | |
| harlowja | k | 21:28 |
| sdake | this runs the aformentioned set_configs.py https://github.com/openstack/kolla/blob/master/docker/base/start.sh#L14 | 21:28 |
| jogam | sdake: sorry but I seem to be lacking something or other, got openstack_release updated in both kolla/etc/kolla/globals.yml and in the root /etc/ folder but the compile is still generating images that are tagged as 3.0.0?? | 21:28 |
| jogam | btw: I am compiling from 2.0.1 tag... | 21:28 |
| harlowja | sdake k, cool, i'll mess around with a few things | 21:29 |
| harlowja | thanks inc0 sdake for the little intro to that and what its doing | 21:29 |
| sdake | set configs.py writes out the /run_command file | 21:29 |
| sdake | we do it this way for security reasons | 21:29 |
| sdake | so containers end up not running as root | 21:29 |
| harlowja | k | 21:29 |
| inc0 | harlowja, any time, let me know if you run into any hurdles | 21:29 |
| sdake | in the case of a container breakout, the hacker doesnt have root privileges on the get go from an internet facing service | 21:29 |
| harlowja | right, fair enough | 21:30 |
| sdake | the run_command file is owned by root | 21:30 |
| sdake | the set_configs.py file is owned by root | 21:30 |
| sdake | othe rfiles are owned by nova or heat or whatever | 21:30 |
| harlowja | makes sense | 21:30 |
| sdake | so basically its not possible to escalate inside the contiainer even if you break into the contianer from a python web f acing app - which is pretty hard to begin with | 21:31 |
| sdake | that said, things like /etc/nova/* | 21:31 |
| sdake | are owned by the nova user | 21:31 |
| sdake | so your database root pwd will be open or all to see | 21:31 |
| sdake | or for the hacker that is | 21:31 |
| sdake | which is not ideal but this is just how unix permissions work | 21:31 |
| harlowja | ya | 21:31 |
| sdake | every other deployment needs to operate in this way | 21:32 |
| sdake | openstack could be smarter about reading the config file then dropping root | 21:32 |
| sdake | but tht is a huge hurdle to tacle | 21:32 |
| harlowja | def | 21:32 |
| sdake | multiply that by the cargo copy paste | 21:32 |
| sdake | * big tent | 21:32 |
| harlowja | yup yup | 21:32 |
| sdake | = not worth my time :) | 21:32 |
| sdake | this is a failure of openstack in generals security model, not of anything related to kolla | 21:33 |
| sdake | any questions? :) | 21:34 |
| sdake | that json file is our ABI | 21:34 |
| sdake | every container follows (for the most part) the same pattern | 21:34 |
| sdake | ceph doesnt drop root | 21:35 |
| sdake | since its a backend service its not super significant | 21:35 |
| sdake | but its not ideal - we will fix it when upstream fixes it | 21:35 |
| harlowja | :) | 21:35 |
| harlowja | that's good enough for now | 21:35 |
| harlowja | gotta mess around with that more | 21:35 |
| sdake | we can't make something that is intended to root drop root | 21:36 |
| sdake | to run as root i mean | 21:36 |
| sdake | ceph is meant to run as root at th emoment | 21:36 |
| sdake | their next releae introduces root dropping | 21:36 |
| openstackgerrit | David Wang proposed openstack/kolla-kubernetes: Updated quickstart doc dependencies section https://review.openstack.org/333591 | 21:38 |
| sdake | jogam ping | 21:39 |
| sdake | jogam run git status | fpate | 21:39 |
| sdake | fpaste if on centos | 21:39 |
| sdake | fi on ubuntu jsut put it in a paste service | 21:39 |
| sdake | inc0 you hae any wizardry way to paste on ubuntu | 21:39 |
| sdake | pastebinit seems to be a favorite of ubuntu fans | 21:41 |
| *** banix has quit IRC | 21:45 | |
| *** banix has joined #openstack-kolla | 21:47 | |
| *** banix has quit IRC | 21:48 | |
| jogam | sdake: HEAD detached at 2.0.1 | 21:49 |
| sdake | pip show kolla | 21:50 |
| sdake | i really wish i could erwrite that documentation in a way that is more clear | 21:51 |
| harlowja | also general question, would all the stuff that is put in the base image (for centos) be ok to split up | 21:51 |
| *** kproskurin has quit IRC | 21:51 | |
| sdake | harlowja how do you mean - please expand | 21:51 |
| jogam | sdake: aaahhh there is goes... Version: 3.0.0.0b2.dev163 | 21:51 |
| sdake | jogam ya - pip remove kolla | 21:51 |
| harlowja | well it seems like some virt stuff is installed even if i'm just asking for say a build of glance | 21:51 |
| sdake | or maybe its uninstall | 21:52 |
| sdake | one of em will get it done | 21:52 |
| *** g3ek has quit IRC | 21:52 | |
| sdake | harlowja never seen that - but for exampe nova-compute installs libvirt | 21:52 |
| harlowja | right | 21:52 |
| sdake | and nova-compute doesn't need to install the 800mg of libvirt | 21:52 |
| sdake | i think that is the biggest offender | 21:53 |
| sdake | that is really up to red hat imo - that is a packaging choice you make with rpms | 21:53 |
| harlowja | # kolla-build glance | 21:53 |
| sdake | with source builds you get exact deps | 21:53 |
| harlowja | just running that | 21:53 |
| sdake | harlowja i have internal demo tomorrow i dont want to disturb my system | 21:53 |
| harlowja | just pulls in weird things :-P | 21:53 |
| harlowja | kk | 21:53 |
| sdake | ya thats rpm packaging nonsense | 21:54 |
| harlowja | ya, stupid rpms | 21:54 |
| harlowja | lol | 21:54 |
| sdake | the best way to make a difference there is complain at rdo community | 21:54 |
| sdake | or involve yourself there | 21:54 |
| sdake | i don't have time to package | 21:54 |
| jogam | sdake: no remove or uninstall - no files found or such, will just re-clone my virtual machine | 21:54 |
| sdake | and spent 9 years of my life doing alot of packaging | 21:54 |
| harlowja | sdake ya, that's why i built anvil which we are getting away from | 21:54 |
| sdake | so i'm done with that ;) | 21:54 |
| harlowja | it autopackages most of the things | 21:54 |
| sdake | jogam there is another approach | 21:54 |
| harlowja | but ya, no thx to packages, lol | 21:55 |
| sdake | jogam change to whereever your python files are stored and rm -rf *kolla* | 21:55 |
| sdake | as in /usr/lib/* | 21:55 |
| jogam | sdake: too late, she´s dead... | 21:56 |
| sdake | i dont use vms for dev | 21:58 |
| sdake | have 5 node cluster at home and 10 node at work | 21:58 |
| jogam | sdake: lucky one I´d say, I am not necessarily getting paid for kolla dev and we are currently evaluating and the VMWare dual host cluster was available and unused... | 21:59 |
| Daviey | Hey, has ceph based nova live migration failing for anyone else on Master? | 21:59 |
| sdake | Daviey i'm not sure anyone has ever tried that tbh :) | 22:00 |
| sdake | Daviey Jeffrey4l__ may have tried it - not sure on ceph | 22:01 |
| *** g3ek has joined #openstack-kolla | 22:01 | |
| sdake | jogam cool - well kolla is pretty cool - but for an eval it will take you probably 4-8 hours to get a hang of how it works | 22:01 |
| sdake | jogam but you should be able to run through the quickstart AIO in about 20 minutes | 22:02 |
| sdake | jogam if I follow the directoins without thinking i can run through the AIO in about 18 minutes - I timed myself at a normal pace | 22:02 |
| jogam | sdake: am at it since last week, got a OS cluster up and running after compiling on 16.04 LTS | 22:02 |
| sdake | (nto including the 15 minute build time of course;) | 22:02 |
| sdake | jogam thtas a real bummer - I think partof the problem is 16.04 | 22:03 |
| Daviey | sdake: i worked on Mitaka | 22:03 |
| Daviey | it* | 22:03 |
| sdake | Daviey well thats grumpy news :( | 22:03 |
| jogam | sdake: true enough, but the warning signs were too small, since part of the intro´s are actually referencing 16.04 | 22:03 |
| sdake | Daviey mind filing a bug - sounds critical to me | 22:03 |
| sdake | jogam ya our docs are targeted at master | 22:04 |
| Daviey | sdake: Yeah.. i'm investigating... I fixed two kolla/ceph bugs over the weekend | 22:04 |
| sdake | jogam but our code we want people to use is not master | 22:04 |
| jogam | hehe :) | 22:04 |
| sdake | Daviey sweet - if you need reviews hit me up :) | 22:04 |
| Daviey | sdake: This needs to land... gttps://review.openstack.org/#/c/334202/ | 22:04 |
| jogam | sdake: I understand that, but all your browsable/googleable docu is for master :) | 22:04 |
| sdake | jogam i'm not quite suree how to fix it in the docs | 22:04 |
| Daviey | err http* | 22:04 |
| sdake | jogam but i was actuallly in the midst of giving it a go right now | 22:05 |
| Daviey | sdake: I haven't been able to reproduce the original issue that my fix reverts. | 22:05 |
| sdake | jogam mind i ask how many nodes your running in total | 22:05 |
| jogam | sdake: here is the config:http://pastebin.com/T6a6Ys7N | 22:09 |
| harlowja | anyway to get a more reliable epel mirror somewhere :-P | 22:10 |
| harlowja | http://paste.openstack.org/show/523665/ | 22:10 |
| sdake | jogam ya your running in 4 node vms, are you planning to deploy openstack only to 4 nodes | 22:10 |
| jogam | sdake: here globals --> http://pastebin.com/ag06Xcrk | 22:10 |
| sdake | harlowja ya I know annoying | 22:10 |
| sdake | that is why we can't make our gate voting | 22:10 |
| harlowja | :( | 22:10 |
| sdake | ine 20 is wrong | 22:10 |
| sdake | you want to deploy 2.0.1 if working from a tag | 22:11 |
| jogam | sdake: for now thats what I got resource wise - if we/the mgmt likes it it´ll be 5 med. perf. machines | 22:11 |
| sdake | or 2.0.2 if working from a git repo | 22:11 |
| sdake | that is stable/mitaka for example | 22:11 |
| sdake | cool | 22:11 |
| jogam | unfortunately not ´actual´ server hardware | 22:11 |
| jogam | sdake: not sure if to me, but all I want is to have something that works :) | 22:13 |
| sdake | well you ge tthat with kolla | 22:13 |
| sdake | possibly other htings like rdo | 22:13 |
| sdake | or even fuel | 22:13 |
| sdake | i think kolla is pretty much "next generation" deployment management though | 22:13 |
| sdake | we learned by watching others with their legacy of 5+ installers :) | 22:14 |
| jogam | sdake: and from what I gathered online you guys are planning to release a ´more´ stable release later this year? | 22:14 |
| *** huikang has quit IRC | 22:14 | |
| jogam | sdake: that is why I am looking at kolla, since if we decide to go OS then I want something that allows me to scale out easily, vs. being a management nightmare | 22:14 |
| sdake | our liberty and mitaka release was and is super stable | 22:15 |
| sdake | i have concerns about newton atm because we have a whole lot on our plate and we seem a bit unfoused | 22:15 |
| jogam | :) that sounds good, once I manage to get it somehow compiled I´ll let you know :) | 22:15 |
| sdake | master moves really fast | 22:16 |
| sdake | and our testing there is insufficient | 22:16 |
| sdake | keep in mind your getting upstream software | 22:17 |
| sdake | and openstack in general is a little buggy here and there | 22:17 |
| sdake | the stuf kolla deploys | 22:17 |
| sdake | and operates | 22:17 |
| wirehead_ | I don’t deploy OpenStack often… but when I do, I deploy master. | 22:17 |
| wirehead_ | :D | 22:17 |
| jogam | sdake: to be honest I am even tempted to go back to master and ubuntu 16.04 LTS and use ceph | 22:18 |
| sdake | harlowja i dont know how your rever going to pull off a bronwfield migration | 22:18 |
| openstackgerrit | Merged openstack/kolla: Remove MariaDB warning from HAproxy https://review.openstack.org/334054 | 22:18 |
| harlowja | sdake thats ok, lol | 22:18 |
| sdake | harlowja but if yuo manage to do so, I'll be there to help :) | 22:18 |
| harlowja | :-P | 22:18 |
| jogam | sdake: the compile and install was rather painless and it worked right away | 22:18 |
| harlowja | life isn't ever simple, ha, thats ok ;) | 22:18 |
| harlowja | it will make a good blog post at least, lol | 22:18 |
| sdake | harlowja have you evaled fuel at all | 22:19 |
| harlowja | nah | 22:19 |
| sdake | me either | 22:20 |
| harlowja | :) | 22:20 |
| sdake | i'd like to hear it from a tech dude | 22:20 |
| harlowja | to much stuff to evail, lol | 22:20 |
| openstackgerrit | Joshua Harlow proposed openstack/kolla: Consistently use sys.exit in one place https://review.openstack.org/334690 | 22:20 |
| sdake | well its really hard to deploy fuel and rdo | 22:20 |
| sdake | have to have real gear | 22:20 |
| *** neilus has quit IRC | 22:21 | |
| *** inc0 has quit IRC | 22:21 | |
| sdake | jogam line 42 is not set | 22:22 |
| *** aNupoisc has quit IRC | 22:22 | |
| sdake | jogam yet line 37 is set | 22:22 |
| jogam | sdake: true, makes no difference though?! | 22:24 |
| sdake | jogam is 100 network owned by you? | 22:24 |
| sdake | this is how you wll access the services like api and whatnot | 22:24 |
| jogam | sdake: line 37 is easily accessible from accesible site wide | 22:24 |
| sdake | cool | 22:25 |
| jogam | sdake: you mean .10, yupp thats the idea | 22:25 |
| sdake | enable_tls: yes | 22:25 |
| sdake | might as well turn it on | 22:25 |
| jogam | sdake: but do not have tls certificates | 22:25 |
| sdake | you can generate them | 22:25 |
| sdake | kolla-ansible certificates | 22:25 |
| jogam | sdake: but they are ´fake´, no? | 22:25 |
| sdake | self-signed | 22:25 |
| jogam | sdake: so whats the point for overhead | 22:26 |
| jogam | or benefit? | 22:26 |
| sdake | right - they encrypt the password and whatnot | 22:26 |
| sdake | but i you dont care aobut that then it doesnn't matter | 22:26 |
| sdake | otherwise password goes across wire in plaintext | 22:26 |
| jogam | sdake: all internal network, behind double firewall and this site is really not that ´concerned´ about security it appears, like global admin passwords and the like | 22:27 |
| sdake | roger | 22:27 |
| sdake | if its intrnal only that sounds ok | 22:27 |
| sdake | i'm pretty hot on security | 22:28 |
| jogam | sdake: whats the performance overhead for TLS, to be honest I did not even look at it since its usually a PITA as well | 22:28 |
| sdake | jogam no idea on performance overhead - haven't measured it | 22:28 |
| jogam | sdake: try to, but not running against windmills here, got already called condescending | 22:28 |
| sdake | jogam the idea of openstack is you horizontally scale by adding nodes as you need | 22:29 |
| *** neilus has joined #openstack-kolla | 22:29 | |
| sdake | someone here? | 22:29 |
| sdake | tbh i think for your app you probably don't need tls | 22:29 |
| sdake | but me personally - i'd always deploy tls | 22:30 |
| sdake | jogam did you get AIO going first? | 22:30 |
| jogam | sdake: that´s why I am biting my teeth on kolla | 22:31 |
| jogam | and openstack | 22:31 |
| sdake | biting yur teeth - never heard that idiom | 22:31 |
| jogam | since the site previously already decided to have to vSphere clusters (2/3 machines each) build and it is starting to become a management nightmare, now they wanted to deploy desktop dual VM machines | 22:31 |
| jogam | sdake: haha good catch, ESL is always going to get me... | 22:32 |
| sdake | i would have never known | 22:32 |
| jogam | but true now that I read it, interesting... | 22:33 |
| sdake | i have herad vsphere is easy to manage - is that not true? | 22:33 |
| *** aNupoisc has joined #openstack-kolla | 22:33 | |
| *** tyrola has joined #openstack-kolla | 22:34 | |
| jogam | true, but now you have to manage two separate clusters on two separate networks | 22:35 |
| jogam | plus the nightmare was going to start with the desktop VM machine ´servers´ | 22:35 |
| sdake | how does that even work :) | 22:35 |
| sdake | you mean remote desktop or whatever its called | 22:35 |
| jogam | sdake: any difference between tools/build.py + options or kolla-build | 22:36 |
| sdake | jogam no difference | 22:36 |
| tyrola | Hey guys, does anyone know what I need to configure in kolla to have gigabit network inside the guests? neutron server and compute nodes have 10gbit network cards. | 22:36 |
| *** ssurana has quit IRC | 22:36 | |
| sdake | tyrola i dont think anything | 22:36 |
| openstackgerrit | Joshua Harlow proposed openstack/kolla: Clean all then yum update https://review.openstack.org/334694 | 22:36 |
| sdake | tyrola you mean you want to rate limit? | 22:36 |
| tyrola | sdake: Mh because I've tested multiple downloads and it seems like the ens3 device has only 100mbit/s | 22:37 |
| tyrola | ethtool isn't very helpful | 22:37 |
| sdake | iv'e never tried benchmarkign on my 10gig switch | 22:37 |
| sdake | i know it makes a world of difference in deploys | 22:38 |
| *** ssurana has joined #openstack-kolla | 22:38 | |
| sdake | and has a big impact on ceph bandwidth | 22:38 |
| sdake | i've actuaally benchmarked ceph | 22:38 |
| sdake | its near wire speed | 22:38 |
| sdake | if yo utake into account all the crap it needs to do to ensure reduandacy | 22:39 |
| sdake | as to your particular question | 22:39 |
| Daviey | sdake: hey, would you mind running a command for me on a compute node? | 22:39 |
| Daviey | $ sudo docker exec -ti nova_libvirt /bin/bash | 22:39 |
| Daviey | (nova-libvirt)[root@amcbk-qb161201 /]# virsh dumpxml $(virsh list | grep instance | head -n1 | awk '{ print $1 }') | grep vnc | 22:39 |
| sdake | Daviey i have mitaka deployed | 22:39 |
| Daviey | perfect | 22:39 |
| sdake | ok moment | 22:39 |
| Daviey | I want to compare mitaka with newton | 22:39 |
| jogam | sdake: no AIO - four nodes right away | 22:40 |
| *** haplo37 has quit IRC | 22:40 | |
| *** g3ek has quit IRC | 22:40 | |
| *** Mr_Broken has quit IRC | 22:40 | |
| jogam | sdake: VM desktops are supposed to host identical VM images accross 5/6 machines that in case of destruction by user we were supposed to ´recover´ from USB HD | 22:41 |
| *** Mr_Broken has joined #openstack-kolla | 22:41 | |
| Daviey | sdake: I need to step away, but i will read scrollback. Ta | 22:41 |
| sdake | Daviey <graphics type='vnc' port='5900' autoport='yes' listen='192.168.1.103' keymap='en-us'> | 22:41 |
| tyrola | sdake: I think it can be really a issue with my 3par storage (I don't use ceph). Its integrated via iscsi. I've tested with this test file "wget http://speedtest.tele2.net/1000GB.zip -O /dev/null". If I use iperf it seems to be much faster. | 22:42 |
| openstackgerrit | Joshua Harlow proposed openstack/kolla: Clean all then yum update https://review.openstack.org/334694 | 22:43 |
| sdake | tyrola are you using master? | 22:43 |
| tyrola | sdake: nope, the mitaka branch but I've merged some icsci stuff 2-3 weeks ago from the master to the branch | 22:44 |
| sdake | nice | 22:44 |
| *** aNupoisc has quit IRC | 22:45 | |
| sdake | tyrola mind providing top 3 pain points of kolla :) | 22:45 |
| sdake | tyrola we are really trying to learn from operators that use our software | 22:45 |
| *** ssurana has quit IRC | 22:45 | |
| sdake | tyrola isdsi is going to be super slow in generlal because it introduces extra copies on the wire | 22:46 |
| tyrola | sdake: I hope I can help you... Maybe there is some kolla issue why the iscsi? does not provide more then 100mbit/s | 22:46 |
| sdake | well ceph for example is fullly meshed | 22:47 |
| sdake | meaning every node can access storage directly from where it needs it | 22:47 |
| tyrola | Each compute node has a dedicated 10gbit port for icsci in a seperate vlan, but sure maybe its too slow. | 22:47 |
| sdake | iscsi is single connection to the storage backend | 22:47 |
| *** g3ek has joined #openstack-kolla | 22:48 | |
| sdake | its just the way the technology works | 22:48 |
| sdake | but i think its worth investigating further | 22:48 |
| *** haplo37 has joined #openstack-kolla | 22:48 | |
| sdake | if iscsi was mae to work in a mesh, then it would work much like ceph | 22:48 |
| sdake | but thatis a big job | 22:49 |
| sdake | as in 2-3 years of effort | 22:49 |
| tyrola | Oh yes, change our storage stuff to fibrechannel would be very nice but also very expensive. We bought the HP 3par just a few weeks ago... | 22:49 |
| sdake | ceph can be used with cots jbod arrays | 22:49 |
| sdake | no need for fibrechnanel | 22:49 |
| sdake | use copper on 10gig | 22:49 |
| sdake | but you bought it so i get it :) | 22:50 |
| jogam | sdake: binary vs source any preference | 22:50 |
| sdake | jogam i prefer either on centos | 22:50 |
| sdake | source definately on ubuntu | 22:50 |
| sdake | source makes smaller images | 22:51 |
| sdake | but changes more often | 22:51 |
| sdake | binary ubuntu is busted atm | 22:51 |
| jogam | sdake: oh ya I remember that was my first compile... | 22:51 |
| jogam | sdake: you suggesting that ubuntu is not likable vs. centos? | 22:52 |
| sdake | persoanlly if you want latest and greatest i'd go with centos | 22:52 |
| jogam | sdake: really... hm! | 22:52 |
| sdake | ubutnu 16.04 will be latet and greatest but the reality is we are 3 months away from an implentation | 22:52 |
| sdake | and that will be newton | 22:52 |
| sdake | red hat has bigger r&d team to implement rdo and centos as well | 22:53 |
| jogam | sdake: true, but my personal experience with red has been pita | 22:53 |
| sdake | yup - i use centos daily ad like it more then i did the fedora experience | 22:53 |
| sdake | but i know ho wto use rpm based distros | 22:54 |
| sdake | but dont know how to use apt based distros | 22:54 |
| sdake | so sounds like you know how to use apt bseed distros so i'd go with ource :) | 22:54 |
| jogam | sdake: apt vs rpm is only a difference of commands no? | 22:55 |
| sdake | the whole distro is laid out differently | 22:55 |
| sdake | i've used fedora for 10+ years | 22:55 |
| sdake | i am intimately familiar with everything about it | 22:55 |
| jogam | my experience is more on the side when something is not working its a lot easier to fix in Debian/Ubuntu than the red(s) | 22:55 |
| sdake | and centos comes from fedora | 22:55 |
| sdake | well systemd has my whole systme in lockdown which I don't like | 22:56 |
| openstackgerrit | David Wang proposed openstack/kolla-kubernetes: Update kolla-kubernetes installation instructions https://review.openstack.org/334616 | 22:57 |
| sdake | but 16.04 has systemd | 22:57 |
| *** Mr_Broken has quit IRC | 22:57 | |
| jogam | sdake: probably wrong perspectives, mine is as a user trying to find a set of commands to fix issue X and that seems to be being a lot better solved for Ubuntu than others... | 22:57 |
| jogam | sdake: had the honor to ´work´ with Oracle Linux for a couple of days here, oh my, lost a lot of hair... | 22:58 |
| *** salv-orl_ has joined #openstack-kolla | 22:59 | |
| sdake | one thing aout fedora - every 6 months it was upgrade time | 22:59 |
| sdake | and that was painful | 22:59 |
| sdake | but you allways were rolling with the latest and greatest | 22:59 |
| sdake | i saw the whole systemd thing evolve over 5 releaswes | 22:59 |
| jogam | hm, my experience with Fedora is mixed/scewed only had to maintain a Fed.5 server for a process control system and that was really painful | 23:01 |
| *** salv-orlando has quit IRC | 23:01 | |
| jogam | what makes systemd so unappetizing for your | 23:02 |
| jogam | -r | 23:02 |
| *** vhosakot has quit IRC | 23:03 | |
| openstackgerrit | Joshua Harlow proposed openstack/kolla: Allow for externally managed configuration https://review.openstack.org/332590 | 23:03 |
| openstackgerrit | Joshua Harlow proposed openstack/kolla: Allow for externally managed configuration https://review.openstack.org/332590 | 23:05 |
| sdake | systemd is fine | 23:06 |
| sdake | uefi hs been an adjustment | 23:06 |
| sdake | the bios the way it was was fine | 23:06 |
| tyrola | sdake: do you think there is a config issue with the libvirt iscsi driver? I've tested the guest networking performance now with different iperf public servers. Normal network seems to be okay (gigabit works). But icsci seems to be limited to 100mbit/s | 23:07 |
| sdake | tyrola is it exactly 100mbit? | 23:07 |
| tyrola | 110~ | 23:07 |
| sdake | sbezverk around? | 23:07 |
| *** Mr_Broken has joined #openstack-kolla | 23:09 | |
| tyrola | I think I will try tomorrow a higher mtu. | 23:12 |
| tyrola | I've checked a document from HP http://h20564.www2.hpe.com/hpsc/doc/public/display?docId=emr_na-c04873990 and they recommend MTU 9000 (jumbo frames) | 23:12 |
| sdake | oh ya jumbo frames area a must | 23:12 |
| sdake | that will make a big improvement | 23:13 |
| sdake | let me see if i can figure how to turn that on | 23:13 |
| tyrola | That would be nice, I will check if we need to change some settings on the 3par and on the switches | 23:14 |
| tyrola | But our networking admin is already out of office, I think I need to wait for tomorrow then | 23:14 |
| sdake | ya you will need to enable umbo frame sbefore we can experiment with getting that going in your openstack deployment | 23:16 |
| sdake | i can telly ou its possible | 23:16 |
| sdake | its documented: http://docs.openstack.org/juno/install-guide/install/apt-debian/content/neutron-network-node.html | 23:16 |
| sdake | what I can't tell you is if it will improve performance | 23:16 |
| *** aNupoisc has joined #openstack-kolla | 23:17 | |
| sdake | tyrola mind i sk how many nodes you have | 23:18 |
| tyrola | currently, 2x controller, 2x networking nodes (neutron) and 4x compute nodes | 23:18 |
| tyrola | the networking and compute nodes does have 2x 10gbit ports (neutron, one internal one external interface) and the compute nodes (storage and network interface) | 23:19 |
| jogam | seem tag 2.0.1 has a couple of ubuntu source compile targets that fail --> http://paste.openstack.org/show/523672/ | 23:19 |
| tyrola | sdake: if you need I can send you full specs, we can continue talking tomorrow if we changed our setup to jumbo frames | 23:21 |
| jogam | tried to recompile mesosphere-base and got keyserver.ubuntu.com timeout --> http://paste.openstack.org/show/523673/ | 23:22 |
| *** Mr_Broken has quit IRC | 23:24 | |
| *** Mr_Broken has joined #openstack-kolla | 23:25 | |
| *** haplo37 has quit IRC | 23:25 | |
| *** jtriley has joined #openstack-kolla | 23:29 | |
| *** haplo37 has joined #openstack-kolla | 23:34 | |
| *** Mr_Broken has quit IRC | 23:38 | |
| *** sacharya has quit IRC | 23:40 | |
| *** tyrola has quit IRC | 23:41 | |
| *** Mr_Broken has joined #openstack-kolla | 23:41 | |
| *** tyrola has joined #openstack-kolla | 23:42 | |
| *** aernhart has quit IRC | 23:46 | |
| *** aernhart has joined #openstack-kolla | 23:48 | |
| *** Mr_Broken has quit IRC | 23:54 | |
| *** Mr_Broken has joined #openstack-kolla | 23:54 | |
| *** tyrola has quit IRC | 23:55 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!