| *** Mr_Broken has quit IRC | 00:01 | |
| *** gfidente|afk has quit IRC | 00:03 | |
| sbezverk | jgriffith you need to run this as root or using sudo | 00:06 |
|---|---|---|
| sbezverk | it tries to write to /etc/kolla but regular user does not have privileges | 00:06 |
| jgriffith | sbezverk: umm... I am running as root | 00:13 |
| jgriffith | sbezverk: note the prompt at the bottom there | 00:14 |
| jgriffith | http://paste.openstack.org/show/524144/ | 00:14 |
| jgriffith | sbezverk: seemed to me it's failing to parse the file (whatever file that is) that's being built no? | 00:15 |
| *** Mr_Broken has joined #openstack-kolla | 00:20 | |
| *** g3ek has quit IRC | 00:25 | |
| *** daneyon_ has quit IRC | 00:25 | |
| openstackgerrit | David Wang proposed openstack/kolla-kubernetes: Allow kubectl to use its own configured endpoint for kube API https://review.openstack.org/335627 | 00:26 |
| *** haplo37 has quit IRC | 00:26 | |
| *** g3ek has joined #openstack-kolla | 00:32 | |
| *** haplo37 has joined #openstack-kolla | 00:32 | |
| *** Mr_Broken has quit IRC | 00:34 | |
| *** Mr_Broken has joined #openstack-kolla | 00:35 | |
| *** Jeffrey4l has joined #openstack-kolla | 00:36 | |
| sbezverk | jgriffith hm, you are right. I missed it.. | 00:37 |
| sbezverk | jgriffith pip list, see if you have kolla, you should not | 00:39 |
| sbezverk | then go to the folder where you clone kolla | 00:40 |
| openstackgerrit | Joshua Harlow proposed openstack/kolla: Allow for externally managed configuration https://review.openstack.org/332590 | 00:40 |
| sbezverk | go to tools directory and run ./kolla-ansible deploy | 00:40 |
| *** zhiwei has joined #openstack-kolla | 00:45 | |
| *** Mr_Broken has quit IRC | 00:47 | |
| *** Mr_Broken has joined #openstack-kolla | 00:47 | |
| *** zhiwei has quit IRC | 00:49 | |
| openstackgerrit | Joshua Harlow proposed openstack/kolla: Fix broken save,list images/dependencies options https://review.openstack.org/335737 | 00:53 |
| openstackgerrit | Joshua Harlow proposed openstack/kolla: Fix broken save,list images/dependencies options https://review.openstack.org/335737 | 00:53 |
| *** zhiwei has joined #openstack-kolla | 00:57 | |
| *** daneyon has joined #openstack-kolla | 00:58 | |
| *** Mr_Broken has quit IRC | 01:01 | |
| *** daneyon has quit IRC | 01:02 | |
| *** phuongnh has joined #openstack-kolla | 01:03 | |
| *** Mr_Broken has joined #openstack-kolla | 01:07 | |
| openstackgerrit | Joshua Harlow proposed openstack/kolla: Consistently use a exception vs sys.exit https://review.openstack.org/334690 | 01:11 |
| *** zhurong has joined #openstack-kolla | 01:16 | |
| *** williamcaban has joined #openstack-kolla | 01:16 | |
| *** Mr_Broken has quit IRC | 01:16 | |
| openstackgerrit | Jack Ning proposed openstack/kolla: Replace horizon default config with custom config https://review.openstack.org/306928 | 01:17 |
| *** Mr_Broken has joined #openstack-kolla | 01:17 | |
| *** haplo37 has quit IRC | 01:18 | |
| *** g3ek has quit IRC | 01:18 | |
| *** g3ek has joined #openstack-kolla | 01:24 | |
| *** haplo37 has joined #openstack-kolla | 01:25 | |
| *** ravig has quit IRC | 01:27 | |
| *** ravig has joined #openstack-kolla | 01:30 | |
| *** Mr_Broken has quit IRC | 01:34 | |
| *** banix has quit IRC | 01:36 | |
| *** Mr_Broken has joined #openstack-kolla | 01:39 | |
| *** ssurana has quit IRC | 01:42 | |
| *** Mr_Broken has quit IRC | 01:47 | |
| *** Mr_Broken has joined #openstack-kolla | 01:47 | |
| *** sdake has joined #openstack-kolla | 01:54 | |
| *** sacharya has joined #openstack-kolla | 01:59 | |
| *** zhurong has quit IRC | 02:00 | |
| *** zhurong has joined #openstack-kolla | 02:00 | |
| *** Mr_Broken has quit IRC | 02:01 | |
| *** sdake has quit IRC | 02:04 | |
| *** sacharya_ has joined #openstack-kolla | 02:12 | |
| *** sacharya has quit IRC | 02:15 | |
| *** Mr_Broken has joined #openstack-kolla | 02:16 | |
| openstackgerrit | Merged openstack/kolla: Add releasenotes for ceilometer https://review.openstack.org/335692 | 02:21 |
| openstackgerrit | Merged openstack/kolla: Fix build for oraclelinux base https://review.openstack.org/329844 | 02:25 |
| *** Mr_Broken has quit IRC | 02:31 | |
| *** ravig has quit IRC | 02:36 | |
| *** ravig has joined #openstack-kolla | 02:37 | |
| *** Mr_Broken has joined #openstack-kolla | 02:38 | |
| *** daneyon has joined #openstack-kolla | 02:46 | |
| *** yuanying has quit IRC | 02:50 | |
| *** daneyon has quit IRC | 02:50 | |
| *** Mr_Broken has quit IRC | 02:51 | |
| *** banix has joined #openstack-kolla | 02:53 | |
| *** coolsvap has joined #openstack-kolla | 02:56 | |
| *** huikang has joined #openstack-kolla | 02:57 | |
| *** Mr_Broken has joined #openstack-kolla | 02:57 | |
| *** harlowja_at_home has joined #openstack-kolla | 03:00 | |
| jgriffith | sbezverk: yes, pip shows kolla installed, I was following the docs here: http://docs.openstack.org/developer/kolla/quickstart.html | 03:00 |
| jgriffith | sbezverk: running again now from kolla/tools/kolla-ansible after pip uninstall | 03:01 |
| *** g3ek has quit IRC | 03:01 | |
| *** Mr_Broken has quit IRC | 03:02 | |
| *** g3ek has joined #openstack-kolla | 03:10 | |
| *** Mech422 has quit IRC | 03:13 | |
| *** Mech422 has joined #openstack-kolla | 03:14 | |
| *** JianqingJiang has joined #openstack-kolla | 03:14 | |
| *** huikang has quit IRC | 03:18 | |
| *** huikang has joined #openstack-kolla | 03:18 | |
| *** tfukushima has joined #openstack-kolla | 03:20 | |
| *** harlowja_at_home has quit IRC | 03:20 | |
| *** Mr_Broken has joined #openstack-kolla | 03:21 | |
| *** harlowja_at_home has joined #openstack-kolla | 03:22 | |
| *** JianqingJiang has quit IRC | 03:29 | |
| *** Mr_Broken has quit IRC | 03:35 | |
| *** tfukushima has quit IRC | 03:37 | |
| *** daneyon has joined #openstack-kolla | 03:40 | |
| *** Mr_Broken has joined #openstack-kolla | 03:40 | |
| *** huikang has quit IRC | 03:44 | |
| *** williamcaban has quit IRC | 03:44 | |
| *** banix has quit IRC | 03:51 | |
| *** yuanying has joined #openstack-kolla | 03:53 | |
| *** haplo37 has quit IRC | 03:55 | |
| *** Mr_Broken has quit IRC | 04:02 | |
| *** haplo37 has joined #openstack-kolla | 04:03 | |
| *** tummy has joined #openstack-kolla | 04:04 | |
| *** tummy has quit IRC | 04:16 | |
| *** dwalsh has joined #openstack-kolla | 04:17 | |
| *** Mr_Broken has joined #openstack-kolla | 04:20 | |
| *** egonzalez has joined #openstack-kolla | 04:25 | |
| *** harlowja_at_home has quit IRC | 04:30 | |
| *** Mr_Broken has quit IRC | 04:32 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla: Add Senlin Docker container https://review.openstack.org/335548 | 04:33 |
| *** daneyon_ has joined #openstack-kolla | 04:34 | |
| *** Mr_Broken has joined #openstack-kolla | 04:39 | |
| *** daneyon_ has quit IRC | 04:39 | |
| *** tummy has joined #openstack-kolla | 04:42 | |
| *** ravig has quit IRC | 04:42 | |
| *** egonzalez has quit IRC | 04:43 | |
| *** sdake has joined #openstack-kolla | 04:49 | |
| *** haplo37 has quit IRC | 04:50 | |
| *** haplo37 has joined #openstack-kolla | 04:50 | |
| *** sdake_ has joined #openstack-kolla | 04:51 | |
| *** g3ek has quit IRC | 04:51 | |
| *** sdake_ has quit IRC | 04:53 | |
| *** sdake has quit IRC | 04:54 | |
| *** g3ek has joined #openstack-kolla | 04:56 | |
| *** salv-orlando has joined #openstack-kolla | 05:00 | |
| *** zhurong has quit IRC | 05:02 | |
| *** salv-orl_ has quit IRC | 05:03 | |
| *** Mr_Broken has quit IRC | 05:05 | |
| *** zhurong has joined #openstack-kolla | 05:08 | |
| *** Mr_Broken has joined #openstack-kolla | 05:10 | |
| *** tfukushima has joined #openstack-kolla | 05:20 | |
| openstackgerrit | Md Nadeem proposed openstack/kolla: Added unit test for volume opertaions https://review.openstack.org/335797 | 05:21 |
| *** sdake has joined #openstack-kolla | 05:24 | |
| *** sacharya_ has quit IRC | 05:27 | |
| *** sacharya has joined #openstack-kolla | 05:27 | |
| *** dwalsh has quit IRC | 05:28 | |
| *** Mr_Broken has quit IRC | 05:32 | |
| *** Mr_Broken has joined #openstack-kolla | 05:33 | |
| *** sdake_ has joined #openstack-kolla | 05:36 | |
| *** mewald has joined #openstack-kolla | 05:36 | |
| *** sdake has quit IRC | 05:37 | |
| *** dwalsh has joined #openstack-kolla | 05:38 | |
| *** rcarrillocruz has quit IRC | 05:46 | |
| *** fragatina has joined #openstack-kolla | 05:46 | |
| coolsvap | sdake_, have a look at https://launchpad.net/kolla/newton/newton-1 | 05:46 |
| sdake_ | lookgs good thanks | 05:47 |
| sdake_ | 169 bbugs fixed | 05:47 |
| sdake_ | quite a few ;) | 05:47 |
| coolsvap | i have marked some bugs with retriage tag which needs to be triaged again or are with no activity from a long time | 05:49 |
| *** Mr_Broken has quit IRC | 05:49 | |
| *** mewald has quit IRC | 05:55 | |
| *** Mr_Broken has joined #openstack-kolla | 05:56 | |
| *** fragatina has quit IRC | 05:57 | |
| *** fragatina has joined #openstack-kolla | 05:57 | |
| *** Mr_Broken has quit IRC | 06:02 | |
| *** harlowja_at_home has joined #openstack-kolla | 06:08 | |
| *** Mr_Broken has joined #openstack-kolla | 06:09 | |
| *** bootsha has joined #openstack-kolla | 06:13 | |
| *** Mr_Broken has quit IRC | 06:13 | |
| *** bootsha has quit IRC | 06:24 | |
| *** bootsha has joined #openstack-kolla | 06:27 | |
| *** sdake_ has quit IRC | 06:30 | |
| *** haplo37 has quit IRC | 06:35 | |
| *** g3ek has quit IRC | 06:38 | |
| *** harlowja_at_home has quit IRC | 06:39 | |
| *** belmoreira has joined #openstack-kolla | 06:40 | |
| *** g3ek has joined #openstack-kolla | 06:40 | |
| *** haplo37 has joined #openstack-kolla | 06:44 | |
| *** Serlex has joined #openstack-kolla | 06:45 | |
| *** Mr_Broken has joined #openstack-kolla | 06:49 | |
| *** sacharya has quit IRC | 06:51 | |
| *** shardy has joined #openstack-kolla | 06:51 | |
| *** bootsha has quit IRC | 06:51 | |
| Mr_Broken | morning guys | 06:54 |
| Mr_Broken | is this up2date? https://blueprints.launchpad.net/kolla/+spec/binary-ubuntu | 06:54 |
| coolsvap | Mr_Broken, i think it should be | 06:57 |
| coolsvap | anything in particular you are looking at? | 06:57 |
| *** b_bezak has joined #openstack-kolla | 06:58 | |
| *** bootsha has joined #openstack-kolla | 07:04 | |
| Mr_Broken | Magnum in Stable/Mitaka on ubuntu Binary ;] | 07:06 |
| Mr_Broken | does it builds from Sources normally and work ? | 07:06 |
| *** mewald has joined #openstack-kolla | 07:07 | |
| *** daneyon_ has joined #openstack-kolla | 07:16 | |
| Mr_Broken | and is kuryr being deployed when I enable_magnum : True ? | 07:18 |
| coolsvap | Mr_Broken, i dont use ubuntu but i maybe Jeffrey4l vikram can confirm about magnum | 07:19 |
| coolsvap | i think magnum should be working on mitaka | 07:19 |
| Mr_Broken | kk, will try from sources thanks | 07:20 |
| *** daneyon_ has quit IRC | 07:21 | |
| *** JianqingJiang has joined #openstack-kolla | 07:24 | |
| *** JianqingJiang has quit IRC | 07:28 | |
| *** g3ek has quit IRC | 07:32 | |
| *** haplo37 has quit IRC | 07:32 | |
| *** haplo37 has joined #openstack-kolla | 07:35 | |
| *** g3ek has joined #openstack-kolla | 07:35 | |
| *** fragatina has quit IRC | 07:36 | |
| apuimedo | Mr_Broken: magnum does not use kuryr yet | 07:41 |
| apuimedo | there's pieces missing in neutron and in kuryr for supporting containers running in VMs | 07:41 |
| Mr_Broken | apuimedo: is there undocumented enable_kuryr : true ? or no way to deploy in stable/mitaka ? | 07:42 |
| Mr_Broken | nvm, I though i saw the image for kuryr being built , but I did run ubuntu from sources now, and its not building Kuryr anymore | 07:43 |
| Mr_Broken | it still tries to build the kuryr docker, when you run ubuntu from binary ... | 07:44 |
| apuimedo | no way yet | 07:44 |
| Mr_Broken | kk, thanks | 07:44 |
| *** rcarrillocruz has joined #openstack-kolla | 07:45 | |
| *** neilus has joined #openstack-kolla | 07:46 | |
| *** mewald has quit IRC | 07:48 | |
| *** fragatina has joined #openstack-kolla | 07:50 | |
| *** neilus has quit IRC | 07:56 | |
| *** neilus has joined #openstack-kolla | 08:00 | |
| *** b_bezak has quit IRC | 08:02 | |
| *** coolsvap is now known as coolsvap_ | 08:02 | |
| *** gfidente has joined #openstack-kolla | 08:04 | |
| *** athomas has joined #openstack-kolla | 08:09 | |
| *** tfukushima has quit IRC | 08:14 | |
| *** tfukushima has joined #openstack-kolla | 08:15 | |
| *** tfukushima has quit IRC | 08:17 | |
| *** tfukushima has joined #openstack-kolla | 08:18 | |
| *** tfukushima has quit IRC | 08:19 | |
| *** tfukushima has joined #openstack-kolla | 08:20 | |
| *** tfukushima has quit IRC | 08:23 | |
| *** g3ek has quit IRC | 08:24 | |
| *** kbyrne has joined #openstack-kolla | 08:25 | |
| *** g3ek has joined #openstack-kolla | 08:33 | |
| *** tfukushima has joined #openstack-kolla | 08:35 | |
| *** b_bezak has joined #openstack-kolla | 08:39 | |
| b_bezak | Hi guys | 08:51 |
| b_bezak | have you seen such error during deploy? | 08:51 |
| b_bezak | http://paste.openstack.org/show/UtMPQuSzvAiHjjoLJtDv/ | 08:51 |
| *** bootsha has quit IRC | 08:52 | |
| b_bezak | I'm using ansible 2.1 | 08:56 |
| b_bezak | docker-py 1.81 | 08:56 |
| b_bezak | docker 1.11.2 | 08:56 |
| b_bezak | and stable/mitaka | 08:57 |
| *** yuanying_ has joined #openstack-kolla | 08:58 | |
| b_bezak | I'll try downgrading ansible first ;) | 08:58 |
| *** apuimedo_ has joined #openstack-kolla | 09:00 | |
| *** kfox1111 has joined #openstack-kolla | 09:00 | |
| *** mandre_ has joined #openstack-kolla | 09:00 | |
| *** fragatina has quit IRC | 09:00 | |
| *** zhiwei_ has joined #openstack-kolla | 09:06 | |
| *** kbyrne has quit IRC | 09:08 | |
| *** tummy has quit IRC | 09:08 | |
| *** yuanying has quit IRC | 09:08 | |
| *** zhiwei has quit IRC | 09:08 | |
| *** diogogmt has quit IRC | 09:08 | |
| *** mark-casey has quit IRC | 09:08 | |
| *** mandre has quit IRC | 09:08 | |
| *** kfox1111_ has quit IRC | 09:08 | |
| *** lmiccini has quit IRC | 09:08 | |
| *** apuimedo has quit IRC | 09:08 | |
| *** nihilifer has quit IRC | 09:08 | |
| *** apuimedo_ is now known as apuimedo | 09:08 | |
| *** mandre_ is now known as mandre | 09:08 | |
| Jeffrey4l | mitaka need ansible<2 b_bezak | 09:10 |
| *** kbyrne has joined #openstack-kolla | 09:11 | |
| *** bootsha has joined #openstack-kolla | 09:12 | |
| b_bezak | yeap, I've downgraded it to 1.9.4 and it worked, thx | 09:12 |
| *** lmiccini has joined #openstack-kolla | 09:13 | |
| b_bezak | however now I have other issue with network interfaces and ansible - http://paste.openstack.org/show/HdPgvUJl8WdRUVzYCuVr/ | 09:14 |
| b_bezak | during precheck and deploy | 09:14 |
| b_bezak | network_interface: "br-int-kolla" | 09:15 |
| b_bezak | storage_interface: "br-ceph" | 09:15 |
| b_bezak | I have such in globals.yml | 09:15 |
| b_bezak | maybe dashes in interface naming causing the issue? | 09:18 |
| *** matrohon has joined #openstack-kolla | 09:20 | |
| *** haplo37 has quit IRC | 09:21 | |
| *** g3ek has quit IRC | 09:21 | |
| *** haplo37 has joined #openstack-kolla | 09:27 | |
| *** g3ek has joined #openstack-kolla | 09:27 | |
| b_bezak | ok, so dashes in interfaces name were the issue here ;) | 09:35 |
| *** zhurong has quit IRC | 09:35 | |
| *** dwalsh has quit IRC | 09:45 | |
| *** nihilifer has joined #openstack-kolla | 09:47 | |
| *** berendt has joined #openstack-kolla | 09:48 | |
| *** daneyon_ has joined #openstack-kolla | 09:59 | |
| *** bootsha has quit IRC | 09:59 | |
| *** daneyon_ has quit IRC | 10:04 | |
| b_bezak | next issue :) with ceph deploy - http://paste.openstack.org/show/zRimXQzfLhuyMhDohiqg/ | 10:04 |
| *** david-lyle has quit IRC | 10:09 | |
| *** david-lyle_ has joined #openstack-kolla | 10:09 | |
| b_bezak | ok I found resolution in kolla ceph guide, however there are issues still with it http://paste.openstack.org/show/bC7CO4WVWcqaemqWbVLM/ | 10:09 |
| *** fragatina has joined #openstack-kolla | 10:10 | |
| *** g3ek has quit IRC | 10:14 | |
| *** haplo37 has quit IRC | 10:15 | |
| *** fragatina has quit IRC | 10:17 | |
| openstackgerrit | Christian Berendt proposed openstack/kolla: Add neutron-vpnaas-agent container https://review.openstack.org/335495 | 10:19 |
| *** g3ek has joined #openstack-kolla | 10:20 | |
| *** haplo37 has joined #openstack-kolla | 10:21 | |
| *** fragatina has joined #openstack-kolla | 10:28 | |
| *** bootsha has joined #openstack-kolla | 10:29 | |
| *** fragatina has quit IRC | 10:32 | |
| *** fragatina has joined #openstack-kolla | 10:33 | |
| *** bootsha has quit IRC | 10:34 | |
| *** tfukushima has quit IRC | 10:39 | |
| *** Detaral has quit IRC | 10:41 | |
| *** aernhart has quit IRC | 10:44 | |
| *** tfukushima has joined #openstack-kolla | 10:45 | |
| *** neilus1 has joined #openstack-kolla | 10:45 | |
| *** aernhart has joined #openstack-kolla | 10:46 | |
| *** aernhart has quit IRC | 10:46 | |
| *** aernhart has joined #openstack-kolla | 10:47 | |
| mdnadeem | Jeffrey4l, ping | 10:49 |
| *** neilus has quit IRC | 10:49 | |
| *** neilus1 has quit IRC | 10:52 | |
| *** neilus has joined #openstack-kolla | 10:52 | |
| Jeffrey4l | mdnadeem, sup? | 10:57 |
| mdnadeem | Jeffrey4l, hi just need to confirm something | 10:58 |
| Jeffrey4l | mdnadeem, whats it? | 10:58 |
| *** salv-orl_ has joined #openstack-kolla | 11:00 | |
| mdnadeem | Jeffrey4l, deploy-task deploy haproxy container on network node and skip the controller nodes , however upgrade-task deploy it on both network and controller node | 11:00 |
| mdnadeem | >> https://github.com/openstack/kolla/blob/master/ansible/roles/haproxy/tasks/deploy.yml#L6 | 11:00 |
| mdnadeem | >> https://github.com/openstack/kolla/blob/master/ansible/roles/haproxy/tasks/upgrade.yml#L19 | 11:01 |
| Jeffrey4l | mdnadeem, sdake report this yesterday. | 11:01 |
| mdnadeem | Is this a bug ? | 11:01 |
| *** phuongnh has quit IRC | 11:01 | |
| Jeffrey4l | no idea for this now :( | 11:01 |
| Jeffrey4l | let me check this. | 11:02 |
| mdnadeem | Jeffrey4l, ok | 11:02 |
| mdnadeem | Jeffrey4l, sdake report it as a bug? | 11:02 |
| Jeffrey4l | mdnadeem, just meet this issue. not file a bug for this now. | 11:03 |
| *** salv-orlando has quit IRC | 11:03 | |
| mdnadeem | Jeffrey4l, then i will file the bug, for tracking | 11:03 |
| Jeffrey4l | mdnadeem, what u really want when deploying haproxy. | 11:03 |
| Jeffrey4l | on control node or network node? | 11:03 |
| *** berendt has quit IRC | 11:04 | |
| mdnadeem | Jeffrey4l, i was upgrading it , then haproxy container fails, while debuggin the issue i come to know this | 11:04 |
| Jeffrey4l | mdnadeem, i think i know how this issue happen. | 11:04 |
| Jeffrey4l | mdnadeem, I just want to know what your expect? | 11:05 |
| *** salv-orl_ has quit IRC | 11:05 | |
| Jeffrey4l | deploy the haproxy on the network is OK to you? | 11:05 |
| mdnadeem | Jeffrey4l, yes | 11:05 |
| Jeffrey4l | OK.. | 11:06 |
| *** ppowell_ has joined #openstack-kolla | 11:07 | |
| *** g3ek has quit IRC | 11:07 | |
| *** mewald has joined #openstack-kolla | 11:07 | |
| mdnadeem | Jeffrey4l, One more thing, have you came across the issue , nova-novncproxy service listen on 0.0.0.0, however i have set my system ip in nova.conf file | 11:08 |
| Jeffrey4l | mdnadeem, which branch are u using? | 11:08 |
| mdnadeem | i face this issue while upgrading from Liberty(1.1.2) to Mitaka(2.0.2) | 11:09 |
| Jeffrey4l | mdnadeem, this should be only happen on liberty branch, i think. | 11:09 |
| Jeffrey4l | and recently, i fixed this issue. | 11:09 |
| *** haplo37 has quit IRC | 11:10 | |
| Jeffrey4l | mdnadeem, could u check this https://review.openstack.org/328882 | 11:10 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Fix the upgrade haproxy failed https://review.openstack.org/335942 | 11:11 |
| mdnadeem | Jeffrey4l, I have face this issue last week, may be now it is solved by your patch | 11:11 |
| Jeffrey4l | mdnadeem, about the haproxy issue. could u try this quick fix? https://review.openstack.org/335942 | 11:11 |
| mdnadeem | Jeffrey4l, i will check it, | 11:12 |
| Jeffrey4l | thanks a lot. | 11:13 |
| *** berendt has joined #openstack-kolla | 11:13 | |
| Jeffrey4l | mdnadeem, maybe u think cherry-pick that PS to the mitaka branch code. then try upgrade from liberty to mitaka. | 11:13 |
| mdnadeem | Jeffrey4l, ok | 11:14 |
| *** g3ek has joined #openstack-kolla | 11:14 | |
| *** haplo37 has joined #openstack-kolla | 11:15 | |
| mdnadeem | Jeffrey4l, now haproxy container upgraded and play proceed further | 11:19 |
| Jeffrey4l | mdnadeem, cool | 11:19 |
| mdnadeem | Jeffrey4l, Thanx | 11:19 |
| mdnadeem | Jeffrey4l, now mariadb container fail to upgrade, may be i have done some mess in my environment, will freshly try it again | 11:20 |
| Jeffrey4l | ok. | 11:20 |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla-kubernetes: POC - Ansible support for Mariadb https://review.openstack.org/334255 | 11:22 |
| *** rhallisey has joined #openstack-kolla | 11:22 | |
| *** mliima has joined #openstack-kolla | 11:28 | |
| b_bezak | ceph problem of mine in stable/mitaka 2.0.2 is still here (I've removed everything and tried again) http://paste.openstack.org/show/SGZcbyipvVpa7i8sOSex/ | 11:30 |
| b_bezak | anybody have similar case? | 11:30 |
| *** tfukushima has quit IRC | 11:30 | |
| Mr_Broken | guys, I can confirm what b_bezak says - when trying to deploy Stable/Mitaka from Ubuntu Sources deploy is blocked on ceph_mon, and than cant remove ceph_mon volume as the documentation states | 11:43 |
| Mr_Broken | anyone have running stable/mitaka with ceph ? | 11:44 |
| *** athomas has quit IRC | 11:44 | |
| b_bezak | ok, I got less then 5% in var ;) | 11:47 |
| b_bezak | error: monitor data filesystem reached concerning levels of available storage space (available: 5% 41462 MB) | 11:47 |
| *** daneyon_ has joined #openstack-kolla | 11:47 | |
| *** zhiwei_ has quit IRC | 11:48 | |
| *** athomas has joined #openstack-kolla | 11:50 | |
| *** daneyon_ has quit IRC | 11:52 | |
| pbourke | b_bezak: is there some existing ceph_mon or container on infra1-kolla that's using the volume? | 11:55 |
| pbourke | hey Jeffrey4l, wondering do you have any thoughts on how to best support external journals using the manual "udev-less" approach? | 11:55 |
| *** sdake has joined #openstack-kolla | 11:55 | |
| pbourke | Jeffrey4l: we can have users supply the json data that is output by find_disks, but that's quite messy to have in a variable | 11:56 |
| *** sdake_ has joined #openstack-kolla | 11:57 | |
| *** kmadac has joined #openstack-kolla | 11:59 | |
| *** g3ek has quit IRC | 12:00 | |
| *** neilus has quit IRC | 12:00 | |
| *** sdake has quit IRC | 12:01 | |
| *** mgoddard_ has joined #openstack-kolla | 12:01 | |
| *** fragatina has quit IRC | 12:01 | |
| *** haplo37 has quit IRC | 12:01 | |
| *** haplo37 has joined #openstack-kolla | 12:03 | |
| *** g3ek has joined #openstack-kolla | 12:03 | |
| *** mgoddard has quit IRC | 12:04 | |
| *** coolsvap_ has quit IRC | 12:05 | |
| *** salv-orlando has joined #openstack-kolla | 12:06 | |
| sdake_ | morning | 12:08 |
| *** ravig has joined #openstack-kolla | 12:08 | |
| *** williamcaban has joined #openstack-kolla | 12:14 | |
| *** salv-orlando has quit IRC | 12:14 | |
| *** stvnoyes has quit IRC | 12:17 | |
| b_bezak | pbourke: I am using infra1 as deployer also. there was ceph-mon in stopped state | 12:19 |
| Jeffrey4l | pbourke, how about read the gpt info from the disk directly? | 12:25 |
| *** jamie_h has joined #openstack-kolla | 12:27 | |
| *** stvnoyes1 has joined #openstack-kolla | 12:32 | |
| *** fragatina has joined #openstack-kolla | 12:34 | |
| sdake_ | Jeffrey4l https://review.openstack.org/#/c/334644/2 | 12:37 |
| patchbot | sdake_: patch 334644 - kolla - Fix upgrades fail to upgrade | 12:37 |
| sdake_ | Jeffrey4l is there a way to pass a warning messsage to ansible rom a module without exiting ansible | 12:37 |
| *** mgoddard_ has quit IRC | 12:38 | |
| sdake_ | this change really needs to go in in some orm or another | 12:38 |
| Jeffrey4l | sdake_, i prefer not simple ignore all the error here. | 12:38 |
| *** mgoddard has joined #openstack-kolla | 12:38 | |
| Jeffrey4l | sdake_, how about add a new param: ignore_errors/forece, when it exsit, ignore all the errors. | 12:39 |
| sdake_ | yes i like this approach | 12:40 |
| sdake_ | we only need this for host=pid mode anyway | 12:40 |
| sdake_ | what could be done is host-pid mode could force it | 12:40 |
| sdake_ | rather pidmode=host | 12:41 |
| Jeffrey4l | sdake_, hmm. it is more tricky. may be one day we will have another container run with --pid=host. | 12:41 |
| sdake_ | anything with --pid=host will exhibit this problem | 12:42 |
| ccesario | Hey Jeffrey4l ... I 'm testing the multipathd ... and it seems I 'm having problem due some missing bindmounts for multipathd ... could you explain me please what this line do ? https://github.com/openstack/kolla/blob/master/ansible/roles/nova/tasks/start_compute.yml#L43 | 12:42 |
| sdake_ | Jeffrey4l could use your hep on ccesario 's q, he been stuck for a week | 12:42 |
| Jeffrey4l | np | 12:42 |
| sdake_ | and i dont knkow what that line does | 12:43 |
| sdake_ | it produces a list of bindmounts | 12:43 |
| sdake_ | from where, no idea | 12:43 |
| Jeffrey4l | sdake_, that depend. only when the container has some state change, like start a new processes in new namespace. right? | 12:43 |
| sdake_ | ccesario sbezverk would know too | 12:43 |
| *** salv-orlando has joined #openstack-kolla | 12:43 | |
| sdake_ | Jeffrey4l dedicated flg is fine by me | 12:44 |
| Jeffrey4l | cool | 12:44 |
| sdake_ | lease leave a review to that effect | 12:44 |
| Jeffrey4l | ok. | 12:44 |
| Jeffrey4l | ccesario, what error you got? | 12:45 |
| Jeffrey4l | volumes: '{{ mounts | reject("equalto", "") | list}}': this line remove the empty line in mounts. | 12:45 |
| Jeffrey4l | because #L32 may be empty. - "{% if enable_iscsi | bool %}iscsi_info:/etc/iscsi{% endif %}" | 12:45 |
| ccesario | Jeffrey4l, due the needed that some cinder drivers usage multipathd , It is needed from nova_compute container multipathd check the storage paths using multipathd -k 'show paths' | 12:48 |
| ccesario | If I install multipathd pack and start the service manually on nova_compute container it works.... | 12:48 |
| Jeffrey4l | ccesario, not familiar with multipath. could u explain more? | 12:48 |
| sdake_ | Jeffrey4l multiplath is like bonding for storage | 12:49 |
| Jeffrey4l | nova_compute need connect the device get from ` multipathd -k 'show paths'`? | 12:50 |
| *** prithiv has joined #openstack-kolla | 12:50 | |
| sdake_ | the problem ccesario has is the socket can't be found | 12:50 |
| Jeffrey4l | so the direct solution is share the socket between the cinder and nova_compute, right? | 12:50 |
| ccesario | Jeffrey4l, yes... the shared socket is the solution | 12:51 |
| Jeffrey4l | ccesario, is that any others solution? without shared the socket? | 12:52 |
| ccesario | Jeffrey4l, http://docs.openstack.org/mitaka/config-reference/block-storage/drivers/ibm-storwize-svc-driver.html this is the driver that I 'm testing .... but other drivers needs multipathd | 12:52 |
| ccesario | Jeffrey4l, currently I get it works installing the multipathd pack on nova_compute using inc0 customize patch | 12:53 |
| ccesario | Jeffrey4l, but the problem is that the service does not startup automaticly on container start | 12:53 |
| Jeffrey4l | ccesario, this solution is bad. Kolla run one process in one container. | 12:54 |
| ccesario | but I don't know if this is it the correct mode to do it... | 12:54 |
| Jeffrey4l | normally,we follow that rule. | 12:54 |
| ccesario | Jeffrey4l, rigth | 12:54 |
| Jeffrey4l | ccesario, is there any possible, the nova_compute can connect the multipathd throught a port rather than a socket? | 12:55 |
| *** g3ek has quit IRC | 12:55 | |
| ccesario | Jeffrey4l, currently I 'm trying do it installing the multipathd on host and try access it from nova_compute container | 12:55 |
| *** haplo37 has quit IRC | 12:55 | |
| ccesario | Jeffrey4l, I don't know.... let me check | 12:55 |
| Jeffrey4l | ccesario, OK. i think this should work. | 12:56 |
| Jeffrey4l | normally, if one service can access from the socket, it should be access through a port with some configuration change. | 12:56 |
| *** haplo37 has joined #openstack-kolla | 12:56 | |
| *** williamcaban has quit IRC | 12:57 | |
| Jeffrey4l | ccesario, what package should be installed to run multipathd? | 12:57 |
| *** g3ek has joined #openstack-kolla | 12:57 | |
| Jeffrey4l | found it. | 12:57 |
| ccesario | RUN apt-get -y install --no-install-recommends multipath-tools sysfsutils | 12:58 |
| *** williamcaban has joined #openstack-kolla | 12:58 | |
| Jeffrey4l | i am using centos 7, so the package name is : device-mapper-multipath i think. | 12:59 |
| ccesario | Jeffrey4l, yep | 13:01 |
| *** williamcaban has quit IRC | 13:02 | |
| *** banix has joined #openstack-kolla | 13:03 | |
| *** williamcaban has joined #openstack-kolla | 13:04 | |
| *** dwalsh has joined #openstack-kolla | 13:05 | |
| *** neilus has joined #openstack-kolla | 13:05 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: WIP Adds ansible support to deploy skydns service https://review.openstack.org/335677 | 13:07 |
| sdake_ | sweet swapnil cleaned up n1 release | 13:07 |
| *** jtriley has joined #openstack-kolla | 13:08 | |
| sdake_ | https://launchpad.net/kolla/newton/newton-1 | 13:08 |
| ccesario | Jeffrey4l, it seems it does not suppport tcp/udp port | 13:08 |
| Jeffrey4l | no idea right now. :( | 13:09 |
| *** sdake has joined #openstack-kolla | 13:10 | |
| ccesario | systemd script from ubuntu pack http://paste.openstack.org/show/524263 | 13:11 |
| *** sdake_ has quit IRC | 13:13 | |
| sdake | ccesario i have presentation in 3 hours of kolla - after that i can help more concretely by examining the source code to figue out what multipathd is doing | 13:13 |
| sdake | ccesario i am a c master | 13:13 |
| sdake | not sure if Jeffrey4l knows systems programming and c or not | 13:14 |
| sdake | for reference, i consider myself pythoon intermediate | 13:14 |
| sdake | beginner->intermediate->expert->master | 13:15 |
| sdake | expert = 10k hours eperience master = 25khrs/experience | 13:15 |
| Serlex | are there docs to enable heka for kolla? | 13:18 |
| ccesario | sdake, sure sure... I 'll continue investigating it.. | 13:21 |
| sdake | Serlex heka is on always | 13:21 |
| sdake | Serlex docker exec heka ls /var/log/kolla | 13:21 |
| ccesario | sdake, whe you finish the presentation ... ping me .. if possible :) | 13:22 |
| sdake | ccesario will you be awake - its about 4 hours out | 13:22 |
| *** williamcaban has quit IRC | 13:23 | |
| Serlex | cheers | 13:24 |
| ccesario | sdake, now 10:20 AM here :) | 13:24 |
| sdake | oh your on the east coast | 13:24 |
| sdake | thought you were not in the us sorry | 13:24 |
| ccesario | :) | 13:24 |
| *** williamcaban has joined #openstack-kolla | 13:28 | |
| *** neilus1 has joined #openstack-kolla | 13:31 | |
| *** neilus1 has quit IRC | 13:31 | |
| *** neilus1 has joined #openstack-kolla | 13:32 | |
| *** neilus1 has quit IRC | 13:32 | |
| *** neilus has quit IRC | 13:32 | |
| *** sacharya has joined #openstack-kolla | 13:34 | |
| *** neilus has joined #openstack-kolla | 13:35 | |
| *** sacharya has quit IRC | 13:39 | |
| *** dwalsh has quit IRC | 13:39 | |
| *** ravig has quit IRC | 13:43 | |
| ccesario | sdake, related to abstract sockets as you mean .. https://www.redhat.com/archives/dm-devel/2013-January/msg00230.html | 13:45 |
| *** haplo37 has quit IRC | 13:47 | |
| *** g3ek has quit IRC | 13:47 | |
| *** ayoung has joined #openstack-kolla | 13:48 | |
| *** Mr_Broken has quit IRC | 13:49 | |
| *** huikang has joined #openstack-kolla | 13:49 | |
| *** kangh_ has joined #openstack-kolla | 13:49 | |
| sdake | ccesario what i'd recommend at this point | 13:49 |
| sdake | (what i wanted to look att he source code was to confirm abstract sockets) | 13:49 |
| *** kangh_ has quit IRC | 13:49 | |
| sdake | is to mail the multipath list | 13:49 |
| sdake | explain to them that you are containerizing multipathd | 13:50 |
| sdake | and docker has no wayt o share abstract socket namespaces | 13:50 |
| sdake | so a file based socket is needed - possibly as an extra option | 13:50 |
| sdake | quick q tho, why not run multipathd in the container? | 13:50 |
| *** neilus has quit IRC | 13:51 | |
| *** dwalsh has joined #openstack-kolla | 13:52 | |
| sdake | ccesario join #docker-dev | 13:52 |
| ccesario | sdake, firstly I 'm testing it on host system, because I suppose that if I run in container I will get the same problems | 13:53 |
| sdake | no if you run multipathd in the container the abstract socket wil be available | 13:53 |
| *** Mr_Broken has joined #openstack-kolla | 13:54 | |
| *** diogogmt has joined #openstack-kolla | 13:54 | |
| ccesario | sdake, hmmmmm | 13:54 |
| sdake | ccesario with caveat it is possible abstract sockets are broken inside containers too | 13:55 |
| *** haplo37 has joined #openstack-kolla | 13:55 | |
| sdake | butu I doubht it | 13:55 |
| sdake | one thing I'd try | 13:55 |
| sdake | is to enable ipc sharing | 13:55 |
| sdake | before going down that path | 13:55 |
| *** Mr_Broken has quit IRC | 13:55 | |
| *** g3ek has joined #openstack-kolla | 13:55 | |
| sdake | https://github.com/openstack/kolla/blob/master/ansible/roles/iscsi/tasks/start.yml#L8 | 13:56 |
| sdake | ccesario can you try that real qucik like | 13:56 |
| ccesario | https://github.com/jumanjihouse/docker-multipathd | 13:58 |
| *** prithiv has quit IRC | 13:59 | |
| ccesario | sdake, let me try do something ... | 13:59 |
| sdake | ccesario the question is really do you want mltipathd running on the host | 13:59 |
| sdake | typically the answer would be yes | 13:59 |
| sdake | just like lvmd | 14:00 |
| sdake | and chrony | 14:00 |
| sdake | so running multipathd in a container isn't really i think what you want | 14:00 |
| sdake | as that above alpine repo does | 14:00 |
| *** inc0 has joined #openstack-kolla | 14:01 | |
| inc0 | good morning | 14:01 |
| ccesario | sdake, I 'm really don't know answer technically your question, In my opion the best way is run multipathd on host and make it available to the containers ... | 14:03 |
| ccesario | inc0, good morning | 14:03 |
| *** dwalsh has quit IRC | 14:03 | |
| *** sdake_ has joined #openstack-kolla | 14:07 | |
| pbourke | hi inc0 | 14:09 |
| *** sdake has quit IRC | 14:11 | |
| inc0 | hey Paul, will you make to midcycle? | 14:11 |
| pbourke | inc0: no it clashes with holidays for me unfortunately | 14:13 |
| *** diogogmt has quit IRC | 14:14 | |
| sdake_ | pbourke hopefully you can make it to summit | 14:14 |
| inc0 | ah shame, we'll have to get beer-over-IP | 14:14 |
| pbourke | inc0: but i'm really hoping to make it to summit | 14:14 |
| inc0 | boip | 14:14 |
| sdake_ | ok guys | 14:14 |
| sdake_ | this upgrade patch | 14:14 |
| sdake_ | here is the best i can do | 14:14 |
| sdake_ | add a flag | 14:15 |
| sdake_ | force-delete | 14:15 |
| sdake_ | only libvirt would use this flag | 14:15 |
| pbourke | inc0: thanks for reviewing my ceph patch. finding it difficult to get something that works cleanly | 14:15 |
| pbourke | inc0: would appreciate any ideas on how you might present data to the bootstrap task manually (i.e. without find_disks) | 14:15 |
| inc0 | sdake_, so issue is you can't delete container with vm in in on btrfs? | 14:16 |
| sdake_ | it is deleted | 14:16 |
| sdake_ | but it leaks btrfs stuff | 14:16 |
| inc0 | pbourke, so I was thinking, in inventory you can specify per host | 14:16 |
| sdake_ | and docker-py returns an exception | 14:16 |
| sdake_ | which blows up ansible | 14:16 |
| inc0 | ceph_osd_devices=/dev/sda2,/dev/sda3 | 14:16 |
| inc0 | like ansible_ssh_password | 14:16 |
| inc0 | do you have traceback? | 14:17 |
| inc0 | of this exception? | 14:17 |
| inc0 | issue is, if we simply silence this exception | 14:17 |
| inc0 | and delete fails for other reason than yours | 14:17 |
| inc0 | it might break more stuff than just what it breaks | 14:18 |
| pbourke | inc0: that's basically what I have, the issue I think is external journals | 14:18 |
| pbourke | inc0: so how can we indicate that some osd devices should use another device for a journal | 14:19 |
| inc0 | hmm, good point | 14:19 |
| inc0 | with this approach you'd need some abomination of data format | 14:19 |
| *** sdake_ is now known as sdake | 14:20 | |
| pbourke | inc0: right, initially I was basically putting the json struct you would get out of find_disks into a variable | 14:21 |
| pbourke | inc0: but that's way too messy | 14:21 |
| inc0 | let's take a look at what ceph-ansible does | 14:21 |
| pbourke | yeah looking at that today also | 14:21 |
| pbourke | one thing of note is that they seem to just use /dev/sdX to refer to disks. So wondering if they're vulnerable to the whole persistent device naming problem | 14:22 |
| inc0 | well /dev/by-id should work just the same | 14:23 |
| pbourke | they're not using that htough | 14:24 |
| pbourke | in their example they have /dev/sdb for example | 14:24 |
| *** zhurong has joined #openstack-kolla | 14:25 | |
| *** zhurong has quit IRC | 14:26 | |
| *** esharao has joined #openstack-kolla | 14:27 | |
| *** ravig has joined #openstack-kolla | 14:28 | |
| *** belmoreira has quit IRC | 14:28 | |
| *** zhurong has joined #openstack-kolla | 14:31 | |
| inc0 | brb | 14:31 |
| *** inc0 has quit IRC | 14:31 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla: Add Senlin Docker container https://review.openstack.org/335548 | 14:37 |
| *** ravig has quit IRC | 14:39 | |
| *** phuongnh has joined #openstack-kolla | 14:40 | |
| *** sacharya has joined #openstack-kolla | 14:40 | |
| *** zhurong has quit IRC | 14:41 | |
| *** haplo37 has quit IRC | 14:43 | |
| *** g3ek has quit IRC | 14:44 | |
| *** ravig has joined #openstack-kolla | 14:44 | |
| *** sacharya has quit IRC | 14:45 | |
| *** g3ek has joined #openstack-kolla | 14:46 | |
| *** haplo37 has joined #openstack-kolla | 14:46 | |
| *** zhurong has joined #openstack-kolla | 14:46 | |
| *** zhurong has quit IRC | 14:47 | |
| *** david-lyle_ is now known as david-lyle | 14:51 | |
| *** sdake_ has joined #openstack-kolla | 14:56 | |
| sdake_ | inc0 i get that issue that is why i suggest a force flag | 14:58 |
| sdake_ | the force flag only willbe used by libvirt | 14:58 |
| sdake_ | otherwise ansible will crater as it currently does if delete fails | 14:58 |
| sdake_ | groan | 14:58 |
| *** sdake has quit IRC | 14:59 | |
| *** sacharya has joined #openstack-kolla | 14:59 | |
| *** ravig has quit IRC | 15:06 | |
| *** Mr_Broken has joined #openstack-kolla | 15:09 | |
| *** vhosakot has joined #openstack-kolla | 15:09 | |
| *** mliima_ has joined #openstack-kolla | 15:09 | |
| *** sdake has joined #openstack-kolla | 15:10 | |
| *** diogogmt has joined #openstack-kolla | 15:12 | |
| *** sdake_ has quit IRC | 15:13 | |
| *** mliima has quit IRC | 15:13 | |
| *** inc0 has joined #openstack-kolla | 15:18 | |
| *** Mr_Broken has quit IRC | 15:18 | |
| inc0 | back, networking problems:/ | 15:18 |
| *** inc0 has quit IRC | 15:23 | |
| sdake | inc0 | 15:24 |
| sdake | Jeffrey4l had suggested a orce flag | 15:24 |
| sdake | force | 15:24 |
| sdake | which would be used by libvirt exclusively | 15:24 |
| sdake | this is pretty much the best that can be done | 15:24 |
| sdake | the alternative is "no upgrades" on centos | 15:24 |
| *** Mr_Broken has joined #openstack-kolla | 15:25 | |
| sdake | fail | 15:25 |
| *** sacharya_ has joined #openstack-kolla | 15:27 | |
| *** b_bezak has quit IRC | 15:28 | |
| *** sacharya has quit IRC | 15:30 | |
| sdake | sean-k-mooney around? | 15:31 |
| *** mliima__ has joined #openstack-kolla | 15:31 | |
| sdake | hey mliima_ | 15:31 |
| sean-k-mooney | sdake: i just joined | 15:33 |
| sdake | cool quick question about performance of ironic | 15:33 |
| sdake | how long does it take to deploy a node | 15:33 |
| *** Mr_Broken has quit IRC | 15:33 | |
| *** thumpba has joined #openstack-kolla | 15:33 | |
| sdake | for example kolla-ansible deploy takes 7-9 minutes for my gear | 15:33 |
| sean-k-mooney | im working form home but just remote desktoped in to my work pc. | 15:33 |
| sean-k-mooney | am not long | 15:34 |
| sean-k-mooney | over a 1G network at home less then a minute | 15:34 |
| sdake | to deploy the entire node? | 15:34 |
| sdake | only 1 minute | 15:34 |
| sean-k-mooney | it depens on the size of the image you are deploying | 15:34 |
| sdake | that doesn't maek any sense | 15:34 |
| sdake | using the installer tool takes me 5-10 minutes just once i kick thiings off | 15:35 |
| *** mliima_ has quit IRC | 15:35 | |
| sean-k-mooney | well it depend on when you start counting | 15:35 |
| sdake | start to finish | 15:35 |
| *** ravig has joined #openstack-kolla | 15:36 | |
| sdake | not including time spent typing commands | 15:36 |
| sdake | i'm tyring to guess how long it will take to deploy kolla once integrated with ironic | 15:36 |
| sean-k-mooney | if you have the node registered in ironic then the deploy is failly fast. i havent done any expcit time test but next time i deploy ill try and time it | 15:37 |
| sdake | node registration is manual process? | 15:37 |
| sean-k-mooney | not entrily. you pass a json/ymal or csv to bifrosts enroll-dynamic playbook which will register all the nodes | 15:38 |
| sean-k-mooney | the file container the ipmi or other credentials to access the node | 15:38 |
| sdake | must the nodes have ipmi for ironic to work? | 15:39 |
| sean-k-mooney | no | 15:39 |
| sean-k-mooney | there are several drivers | 15:39 |
| sdake | cool because i dont have ipmi | 15:39 |
| sean-k-mooney | what i spend yesterday doing was getting the ssh driver to work so that i could test/dev with vms | 15:39 |
| sdake | i'd like to try on real metal | 15:40 |
| *** Mr_Broken has joined #openstack-kolla | 15:40 | |
| sdake | for real metal you need ipmi? | 15:40 |
| sean-k-mooney | i have reworked my initall bifrost patch to include ubuntu support. ill try and documet so of the vm based setup when i upload it. | 15:40 |
| sean-k-mooney | sdake: if you have intel nucs there is an amt driver | 15:41 |
| sdake | what is nucs | 15:41 |
| *** inc0 has joined #openstack-kolla | 15:41 | |
| sdake | [08:24:03] <sdake>inc0 | 15:41 |
| sdake | [08:24:13] <sdake>Jeffrey4l had suggested a orce flag | 15:41 |
| sean-k-mooney | there is also a wake on lan driver and some propritory dirvers for diferent vendors | 15:41 |
| sdake | [08:24:14] <sdake>force | 15:41 |
| sdake | [08:24:22] <sdake>which would be used by libvirt exclusively | 15:41 |
| sdake | [08:24:31] <sdake>this is pretty much the best that can be done | 15:41 |
| sdake | [08:24:40] <sdake>the alternative is "no upgrades" on centos | 15:41 |
| sdake | wake on lan I think I have that | 15:41 |
| inc0 | sdake, can you please show me traceback? | 15:42 |
| inc0 | I don't like adding flag because we can't figure out actual problem | 15:42 |
| sean-k-mooney | sdake http://www.intel.com/content/www/us/en/nuc/products-overview.html | 15:42 |
| inc0 | also not having upgrades on centos is not true | 15:43 |
| inc0 | you can have upgrades if you use aufs | 15:43 |
| inc0 | we might not have upgrades on btrfs but that's not half as bad as you make it | 15:43 |
| *** mummer has joined #openstack-kolla | 15:43 | |
| inc0 | sdake, can you change your storage driver to aufs if you need it fast? and give us some time to figure out actual problem? | 15:44 |
| *** pbourke has quit IRC | 15:44 | |
| *** pbourke has joined #openstack-kolla | 15:44 | |
| sdake | nothiing worse then having 4 ligthers and all of them being dead | 15:45 |
| sean-k-mooney | inc0: the only issue with changing the storage direve is all existing container and images will be un reachable | 15:45 |
| sdake | inc0 aufs is not on centos dude | 15:45 |
| inc0 | there is no aufs on centos at all? | 15:45 |
| sean-k-mooney | sdake: you could use overlayfs | 15:45 |
| inc0 | sdake, I'm setting up new env, I'll make it btrfs | 15:46 |
| inc0 | and see if ubuntu 16.04 works | 15:46 |
| *** jmccarthy has joined #openstack-kolla | 15:46 | |
| sean-k-mooney | inc0: i use btrfs by default when i deploy | 15:46 |
| *** jmccarthy has left #openstack-kolla | 15:46 | |
| sean-k-mooney | i have a 16.04 based deployment using it now | 15:46 |
| inc0 | sean-k-mooney, I used aufs out of laziness | 15:47 |
| sdake | i am not recommending overlayfs for centos at this time - too dangerous | 15:47 |
| inc0 | but since you have btrfs | 15:47 |
| inc0 | can you test one thing for me? | 15:47 |
| inc0 | setup stable/liberty | 15:47 |
| inc0 | spawn a vm | 15:47 |
| inc0 | upgrade to mitaka | 15:47 |
| sdake | ya tht is what fails the upgrade | 15:47 |
| sdake | silent errors better then no upgrades | 15:48 |
| sdake | and its more like a warning | 15:48 |
| inc0 | I prefere no errors than silent errors | 15:48 |
| sean-k-mooney | ah i can do it in a vm but currently the setup with btrfs is hosting all of my dev vms so i cant tear it down easily | 15:48 |
| *** harlowja_at_home has joined #openstack-kolla | 15:48 | |
| sdake | sean-k-mooney it has to be on bare metal for upgrade to work atall | 15:49 |
| inc0 | sdake, why? | 15:49 |
| *** harlowja_at_home has quit IRC | 15:49 | |
| sdake | something about the kvm processe being killed iirc | 15:49 |
| inc0 | you can run kvm in vm | 15:49 |
| inc0 | ... | 15:49 |
| inc0 | if your host have nested virt enabled | 15:49 |
| sean-k-mooney | yes i use nested virt in my setup | 15:50 |
| sdake | well you debugged it - i dont know the details | 15:50 |
| inc0 | it's ok for vms | 15:50 |
| *** salv-orlando has quit IRC | 15:50 | |
| sdake | inc0 can you explain exactly what you propose to handle this scenario on centos? | 15:51 |
| sdake | inc0 i see a -1 vote thrre but no action specified | 15:51 |
| inc0 | sdake, let's start by you providing me a traceback | 15:51 |
| *** Mr_Broken has quit IRC | 15:51 | |
| inc0 | we need to get to the bottom of the problem | 15:52 |
| sdake | i have already debugged the problem thourougly | 15:52 |
| inc0 | try:except:pass is horrible thing | 15:52 |
| inc0 | so what's wrong? | 15:52 |
| inc0 | really? | 15:52 |
| sdake | the problem is docker has references to btrfs | 15:52 |
| sdake | and docker will not delete the btrfs data | 15:52 |
| sdake | the references come from libvirt because pid=host | 15:52 |
| sdake | libvirt is running in those btrfs data partitions | 15:53 |
| sdake | and cnanot be deleted | 15:53 |
| sdake | it would destroy the virt runtime | 15:53 |
| inc0 | so we can't upgrade libvirt with btrfs | 15:53 |
| inc0 | ? | 15:53 |
| sdake | you can | 15:53 |
| sdake | youcan run upgrade for each compute node repeatidly | 15:53 |
| sdake | thi will cause each compute node to generate a backtrace | 15:53 |
| sdake | the next upgrade will start libvirt on that node | 15:53 |
| sdake | actually it doesn't even generatee a backtrace | 15:54 |
| inc0 | so what happends with old libvirt container? | 15:54 |
| inc0 | you just said it's not deleted | 15:54 |
| sdake | but ya if you have 64 nodes, running ugprade 64 times is not good | 15:54 |
| sdake | it is deleted frrom docker's registration | 15:54 |
| sdake | the storage itself is not deleted | 15:54 |
| *** fragatina has quit IRC | 15:54 | |
| pbourke | Mech422: with a small mod your patch works for me! | 15:55 |
| *** fragatina has joined #openstack-kolla | 15:55 | |
| *** Mr_Broken has joined #openstack-kolla | 15:55 | |
| sdake | in essence docker ps no longer shows the container (as it has been killed and removed) | 15:56 |
| sdake | but the remove operation returns a 500 error from docker because docker-egnine doesn't handle the case of pid=host with btrfs | 15:56 |
| sdake | if I was docker and fixing the problem, I'd just mask the error as well | 15:56 |
| sean-k-mooney | sdake: is the contaier show with docker ps -a | 15:57 |
| inc0 | do you have stale data in /var/luib/docker? | 15:57 |
| inc0 | that's first | 15:57 |
| inc0 | second, this seems like docker bug | 15:57 |
| sdake | it is a docker bug | 15:57 |
| inc0 | did you report it? | 15:57 |
| sdake | yes i reported it | 15:57 |
| sdake | nonetheless we have no upgrade avaialble on centos | 15:58 |
| inc0 | ok, another idea | 15:58 |
| inc0 | http://docs.ansible.com/ansible/playbooks_error_handling.html | 15:58 |
| inc0 | ignoring failed commands | 15:59 |
| inc0 | or even better | 15:59 |
| sdake | here is the deal guys - if you vote -1 please provide guidince on how to fix the problem - a -1 vote with no solution is just like a -2 vote | 15:59 |
| inc0 | you're saying that second time you run this it works? | 15:59 |
| sdake | i saw that | 15:59 |
| sdake | it wors only for one node | 15:59 |
| inc0 | hold on | 15:59 |
| *** mgoddard has quit IRC | 15:59 | |
| inc0 | so if you run second time for same node, it passes? | 15:59 |
| sdake | yes because docker deleted libvirt | 15:59 |
| sdake | and the upgrade runs libvirt again | 15:59 |
| inc0 | add retry to ansible task | 16:00 |
| inc0 | this one | 16:00 |
| *** papacz has quit IRC | 16:00 | |
| *** dims has quit IRC | 16:00 | |
| inc0 | and see if that fixes your issue | 16:00 |
| sdake | how does that not mask the error? | 16:01 |
| inc0 | it will still show the error | 16:01 |
| inc0 | just try again | 16:01 |
| inc0 | and it won't be global | 16:01 |
| sdake | i undersstand global | 16:01 |
| inc0 | so we wont mask EVERY error EVERYWHERE | 16:01 |
| sdake | but i can try the retry | 16:01 |
| inc0 | and if it's different error, persistant one | 16:01 |
| sdake | i already ageed to use jeffrey's method with a force flag | 16:01 |
| inc0 | it will fail 5 retries and fail alltogether | 16:02 |
| inc0 | this method is better | 16:02 |
| sdake | ya souns better | 16:02 |
| sdake | thanks for providing feedback - needd those with -1 votes pls :) | 16:02 |
| *** ravig has quit IRC | 16:03 | |
| inc0 | I -1 horrible solution (sorry for being blunt), it takes more time to find out good one | 16:03 |
| inc0 | also, hey I'm here, just discuss stuff | 16:03 |
| *** Mr_Broken has quit IRC | 16:03 | |
| inc0 | anyway | 16:03 |
| sdake | i've been working at it for going on 2 days ;) | 16:03 |
| sdake | its not a horrible solution | 16:03 |
| sdake | woudl be better without global effect | 16:04 |
| inc0 | it is horrible soluiton, if something is broken with your docker, this will simply go on | 16:04 |
| sdake | if retry prints an error people will be confused | 16:04 |
| inc0 | no questions asked or error logged | 16:04 |
| inc0 | unless they smack ctrl+c in panic, they'll be ok | 16:04 |
| inc0 | before confusion hits it will pass | 16:05 |
| *** matrohon has quit IRC | 16:05 | |
| *** dims has joined #openstack-kolla | 16:06 | |
| sdake | dims | 16:06 |
| sdake | have a moment | 16:06 |
| *** ravig has joined #openstack-kolla | 16:06 | |
| *** Jeffrey4l has quit IRC | 16:08 | |
| *** Mr_Broken has joined #openstack-kolla | 16:12 | |
| *** fragatina has quit IRC | 16:15 | |
| openstackgerrit | Nic Roland proposed openstack/kolla-kubernetes: Use pbr to get version instead of hard coding in CLI https://review.openstack.org/336103 | 16:15 |
| *** sdake_ has joined #openstack-kolla | 16:15 | |
| *** mummer has quit IRC | 16:16 | |
| *** Serlex has quit IRC | 16:17 | |
| sdake_ | inc0 what should be used in the until line? | 16:18 |
| inc0 | can you just use retries: 3 | 16:18 |
| inc0 | I think that was syntax | 16:18 |
| inc0 | let me check | 16:18 |
| sdake_ | i can't right now | 16:18 |
| sdake_ | but in 1 hour i can | 16:18 |
| sdake_ | have demo in about 7 minutes | 16:18 |
| *** sdake has quit IRC | 16:18 | |
| inc0 | sdake_, retries: 3 | 16:19 |
| inc0 | have fun | 16:19 |
| sdake_ | oh ya fun fun :) | 16:19 |
| *** sdake_ is now known as sdake | 16:23 | |
| *** Mr_Broken has quit IRC | 16:24 | |
| *** Mr_Broken has joined #openstack-kolla | 16:24 | |
| *** ravig has quit IRC | 16:26 | |
| *** esharao has quit IRC | 16:26 | |
| *** ravig has joined #openstack-kolla | 16:26 | |
| *** Mr_Broke_ has joined #openstack-kolla | 16:30 | |
| *** Mr_Broken has quit IRC | 16:32 | |
| openstackgerrit | Christian Berendt proposed openstack/kolla: Add neutron-vpnaas-agent container https://review.openstack.org/335495 | 16:33 |
| *** williamcaban has quit IRC | 16:41 | |
| *** ssurana has joined #openstack-kolla | 16:43 | |
| *** g3ek has quit IRC | 16:45 | |
| *** haplo37 has quit IRC | 16:46 | |
| *** g3ek has joined #openstack-kolla | 16:47 | |
| *** haplo37 has joined #openstack-kolla | 16:47 | |
| *** salv-orlando has joined #openstack-kolla | 16:51 | |
| *** salv-orl_ has joined #openstack-kolla | 16:57 | |
| *** inc0 has quit IRC | 16:57 | |
| *** zhiwei has joined #openstack-kolla | 16:57 | |
| *** inc0 has joined #openstack-kolla | 16:57 | |
| *** salv-orlando has quit IRC | 16:58 | |
| *** zhiwei has quit IRC | 17:01 | |
| *** salv-orl_ has quit IRC | 17:02 | |
| *** ravig has quit IRC | 17:03 | |
| *** ravig has joined #openstack-kolla | 17:04 | |
| *** daneyon_ has joined #openstack-kolla | 17:12 | |
| openstackgerrit | Merged openstack/kolla-kubernetes: Use pbr to get version instead of hard coding in CLI https://review.openstack.org/336103 | 17:15 |
| *** daneyon_ has quit IRC | 17:17 | |
| *** ravig has quit IRC | 17:17 | |
| *** ravig has joined #openstack-kolla | 17:22 | |
| openstackgerrit | Merged openstack/kolla-kubernetes: Updated quickstart doc dependencies section https://review.openstack.org/333591 | 17:24 |
| *** salv-orlando has joined #openstack-kolla | 17:29 | |
| wirehead_ | rhallisey: what do ya want to do about the parallization issue in https://review.openstack.org/#/c/330777/ ? | 17:33 |
| patchbot | wirehead_: patch 330777 - kolla-kubernetes - Fix keystone bootstrap failure | 17:33 |
| rhallisey | wirehead_, what did you say was the underlying issue? | 17:34 |
| *** salv-orlando has quit IRC | 17:34 | |
| wirehead_ | rhallisey: Both containers were trying to mount the same log drive at the exact same time, because ncpu = 2. | 17:34 |
| rhallisey | ah that's right | 17:35 |
| *** aernhart has quit IRC | 17:36 | |
| *** aernhart has joined #openstack-kolla | 17:37 | |
| rhallisey | wirehead_, can you create a bug describing the issue. This issue will affect more than just keystone | 17:39 |
| wirehead_ | Sure. | 17:39 |
| wirehead_ | I’ll push a new version to address the other issues. | 17:40 |
| rhallisey | wirehead_, awesome! | 17:40 |
| wirehead_ | rhallisey: I kinda want to switch logging over to just go to stdout and let Kubernetes manage collection of logs. | 17:41 |
| wirehead_ | rhallisey: The downside is that makes us more of a unicorn. | 17:41 |
| rhallisey | well what if we had a side car container scoop them up from std out | 17:41 |
| *** williamcaban has joined #openstack-kolla | 17:42 | |
| wirehead_ | rhallisey: You could, but assuming that all true OpenStack operators use something akin to the ELK pipeline, you’d just configure kube to point at your chosen ELK instance | 17:43 |
| rhallisey | we can log to stdout for now. When we move to centralized logging it's something that we can come back to | 17:43 |
| *** sogabe has quit IRC | 17:44 | |
| *** gfidente has quit IRC | 17:44 | |
| *** athomas has quit IRC | 17:44 | |
| *** haplo37 has quit IRC | 17:44 | |
| *** mewald has quit IRC | 17:44 | |
| *** g3ek has quit IRC | 17:45 | |
| openstackgerrit | sean mooney proposed openstack/kolla: [WIP] experimental bifrost support https://review.openstack.org/319703 | 17:45 |
| *** sdake has quit IRC | 17:45 | |
| *** ravig has quit IRC | 17:46 | |
| *** williamcaban has quit IRC | 17:46 | |
| *** g3ek has joined #openstack-kolla | 17:46 | |
| *** haplo37 has joined #openstack-kolla | 17:47 | |
| sean-k-mooney | hi am general question. show i add the ansible code to deploy a the bifrost container in the same commit that adds it or can i add that as a different commit | 17:48 |
| inc0 | sean-k-mooney, make it 2 commits | 17:50 |
| inc0 | one with container other one with ansible stuff | 17:50 |
| sean-k-mooney | inc0: ya that was what i was leaning towrods | 17:51 |
| inc0 | more commits == easier to review | 17:51 |
| inc0 | smaller ones | 17:51 |
| inc0 | so quicker merge | 17:51 |
| sean-k-mooney | yep i am resonable far in terms of the bifrost container image. but i have to start on the ansible next | 17:52 |
| sean-k-mooney | inc0: out of interest what a good reference for how to include files at runtime in a contaer like the config files? | 17:54 |
| inc0 | sean-k-mooney, https://github.com/openstack/kolla/blob/master/ansible/roles/nova/templates/nova-ssh.json.j2 | 17:55 |
| inc0 | config_files will get copier from source to dest | 17:55 |
| inc0 | source is generally bindmounted dir | 17:56 |
| sean-k-mooney | inc0: thanks. one of the things im not quite sure how to do is to copy an ssh key into the bifrost imate. i could do it via a bind mount but i have a similar issue for external os images | 17:57 |
| *** salv-orlando has joined #openstack-kolla | 17:57 | |
| sean-k-mooney | /imate/container/ | 17:58 |
| inc0 | hmm | 17:58 |
| inc0 | after it's already run right? | 17:58 |
| inc0 | so I'd create volume and bind this volume | 17:58 |
| sean-k-mooney | yes | 17:58 |
| sean-k-mooney | ah ok | 17:58 |
| inc0 | and to place file inside named volume | 17:59 |
| inc0 | depends how it works in real life really | 17:59 |
| sean-k-mooney | without just going into /var/lib/docker/... can you easily copy files into a docker volumn | 18:00 |
| sean-k-mooney | i have never tried that | 18:00 |
| inc0 | well....it's either copy it to /var/lib/docker.. or docker exec with stuff | 18:01 |
| sean-k-mooney | i mean i could just use docker cp to copy the file into the container i guess just not sure if the kolla docker module supports it | 18:01 |
| inc0 | if it doesn't, it's easy add | 18:01 |
| inc0 | however if you have file during deploy | 18:02 |
| inc0 | you can use method I described higher up | 18:02 |
| inc0 | nova-ssh actually copies keys | 18:02 |
| sean-k-mooney | ok ill look at both. | 18:02 |
| sean-k-mooney | there are two models i want to support for images | 18:03 |
| sean-k-mooney | one bifrost builds the image for you, and second the bring your own modle | 18:03 |
| sean-k-mooney | in the byoi case it would be expected you have the image at deploy time | 18:04 |
| sean-k-mooney | in the bifrost build case bifrost will build it at deploy time | 18:04 |
| sean-k-mooney | for the ssh key it would be added to the contaier at deploy time also. | 18:05 |
| *** salv-orlando has quit IRC | 18:05 | |
| *** salv-orlando has joined #openstack-kolla | 18:05 | |
| inc0 | so kolla docker won't really help you after deploy | 18:06 |
| inc0 | so if you want to manually add file to container well...you need to do manually;) | 18:07 |
| *** Mr_Broke_ has quit IRC | 18:07 | |
| sean-k-mooney | this will be part of the kolla host playbook. i will probaly use the command module to do a docker cp to the container or named volume | 18:08 |
| sean-k-mooney | on of the input paramater when you want to supply an image or ssh key to use will be the path to the file | 18:10 |
| inc0 | hold on | 18:10 |
| inc0 | kolla-host will be deploying docker and stuff | 18:10 |
| inc0 | or I'm mistaken | 18:11 |
| inc0 | well, make a PoC Sean, we'll see where that gets us | 18:11 |
| sean-k-mooney | yes it will. | 18:11 |
| sean-k-mooney | what bifrost is intended to do is install the os on the server. the kolla host would install docker... | 18:12 |
| inc0 | yeah, you're right | 18:12 |
| *** dwalsh has joined #openstack-kolla | 18:14 | |
| openstackgerrit | Ken Wronkiewicz proposed openstack/kolla-kubernetes: Fix keystone bootstrap failure https://review.openstack.org/330777 | 18:14 |
| *** dwalsh has quit IRC | 18:15 | |
| sean-k-mooney | i would really like to get to the point where our quickstart guide is basically install ansible then run play book to prep build host, then build images, then run playbook to deploy bifrost and install os then run current playbook to deploy containers | 18:17 |
| *** thumpba has quit IRC | 18:18 | |
| sean-k-mooney | there is a little inception in preping the build host as well something has to install ansible but i dont think everything need to be automated. but if we can reduce it to a few commands that would be nice | 18:19 |
| *** ravig has joined #openstack-kolla | 18:26 | |
| *** thumpba has joined #openstack-kolla | 18:29 | |
| *** Mr_Broken has joined #openstack-kolla | 18:30 | |
| *** Mr_Broken has quit IRC | 18:33 | |
| *** thumpba has quit IRC | 18:34 | |
| *** Mr_Broken has joined #openstack-kolla | 18:34 | |
| *** thumpba has joined #openstack-kolla | 18:35 | |
| *** thumpba has quit IRC | 18:35 | |
| *** thumpba has joined #openstack-kolla | 18:35 | |
| *** g3ek has quit IRC | 18:41 | |
| *** shardy has quit IRC | 18:41 | |
| *** ravig has quit IRC | 18:43 | |
| *** haplo37 has quit IRC | 18:46 | |
| *** haplo37 has joined #openstack-kolla | 18:47 | |
| *** g3ek has joined #openstack-kolla | 18:50 | |
| *** ravig has joined #openstack-kolla | 18:51 | |
| *** ccesario has quit IRC | 18:56 | |
| *** aernhart has quit IRC | 18:58 | |
| *** aernhart has joined #openstack-kolla | 18:59 | |
| *** daneyon_ has joined #openstack-kolla | 19:01 | |
| *** ravig has quit IRC | 19:01 | |
| *** Mr_Broken has quit IRC | 19:03 | |
| openstackgerrit | Ken Wronkiewicz proposed openstack/kolla-kubernetes: Document briefly how services in k8s work and why 0.0.0.0 is OK. https://review.openstack.org/333636 | 19:04 |
| *** Mr_Broken has joined #openstack-kolla | 19:04 | |
| *** daneyon_ has quit IRC | 19:05 | |
| *** inc0 has quit IRC | 19:09 | |
| *** Mr_Broken has quit IRC | 19:19 | |
| *** Mr_Broken has joined #openstack-kolla | 19:19 | |
| *** zhiwei has joined #openstack-kolla | 19:24 | |
| *** sacharya_ has quit IRC | 19:26 | |
| *** banix has quit IRC | 19:26 | |
| *** sacharya has joined #openstack-kolla | 19:26 | |
| *** inc0 has joined #openstack-kolla | 19:28 | |
| *** zhiwei has quit IRC | 19:28 | |
| *** ravig has joined #openstack-kolla | 19:32 | |
| openstackgerrit | David Wang proposed openstack/kolla-kubernetes: Update kolla-kubernetes installation instructions https://review.openstack.org/334616 | 19:33 |
| *** ravig has quit IRC | 19:35 | |
| *** ravig has joined #openstack-kolla | 19:36 | |
| openstackgerrit | Christian Berendt proposed openstack/kolla: Add neutron-vpnaas-agent container https://review.openstack.org/335495 | 19:37 |
| *** ravig has quit IRC | 19:41 | |
| openstackgerrit | Merged openstack/kolla-kubernetes: Fix keystone bootstrap failure https://review.openstack.org/330777 | 19:41 |
| *** williamcaban has joined #openstack-kolla | 19:44 | |
| *** g3ek has quit IRC | 19:45 | |
| *** mliima__ has quit IRC | 19:46 | |
| *** ravig has joined #openstack-kolla | 19:46 | |
| *** haplo37 has quit IRC | 19:46 | |
| *** stvnoyes1 has quit IRC | 19:49 | |
| *** williamcaban has quit IRC | 19:49 | |
| *** haplo37 has joined #openstack-kolla | 19:52 | |
| *** Mr_Broken has quit IRC | 19:52 | |
| *** g3ek has joined #openstack-kolla | 19:52 | |
| *** ppowell_ has quit IRC | 19:54 | |
| *** stvnoyes has joined #openstack-kolla | 19:56 | |
| *** inc0 has quit IRC | 20:01 | |
| *** sbezverk has quit IRC | 20:01 | |
| *** fragatina has joined #openstack-kolla | 20:03 | |
| *** Mr_Broken has joined #openstack-kolla | 20:05 | |
| *** ravig has quit IRC | 20:13 | |
| *** banix has joined #openstack-kolla | 20:35 | |
| *** jtriley has quit IRC | 20:36 | |
| *** bootsha has joined #openstack-kolla | 20:39 | |
| *** thumpba has quit IRC | 20:40 | |
| *** bootsha has quit IRC | 20:45 | |
| *** Mr_Broken has quit IRC | 20:49 | |
| *** daneyon_ has joined #openstack-kolla | 20:49 | |
| *** ravig has joined #openstack-kolla | 20:50 | |
| *** daneyon_ has quit IRC | 20:53 | |
| *** ravig has quit IRC | 20:56 | |
| *** ravig has joined #openstack-kolla | 20:56 | |
| *** ssurana has quit IRC | 21:00 | |
| *** bootsha has joined #openstack-kolla | 21:01 | |
| *** ssurana has joined #openstack-kolla | 21:01 | |
| *** ravig has quit IRC | 21:07 | |
| *** Mr_Broken has joined #openstack-kolla | 21:07 | |
| *** mhayden has joined #openstack-kolla | 21:09 | |
| *** ravig has joined #openstack-kolla | 21:17 | |
| *** ravig has quit IRC | 21:19 | |
| *** Mr_Broke_ has joined #openstack-kolla | 21:22 | |
| *** Mr_Broken has quit IRC | 21:23 | |
| *** aernhart has quit IRC | 21:23 | |
| *** ravig has joined #openstack-kolla | 21:24 | |
| *** aernhart has joined #openstack-kolla | 21:24 | |
| *** ravig has quit IRC | 21:27 | |
| openstackgerrit | Suhail Syed proposed openstack/kolla: Fixed typo in all.yml https://review.openstack.org/336294 | 21:29 |
| *** kbyrne has quit IRC | 21:29 | |
| *** kbyrne has joined #openstack-kolla | 21:31 | |
| *** banix has quit IRC | 21:32 | |
| *** banix has joined #openstack-kolla | 21:33 | |
| *** ayoung has quit IRC | 21:34 | |
| *** Mr_Broke_ has quit IRC | 21:34 | |
| *** Mr_Broken has joined #openstack-kolla | 21:35 | |
| *** diogogmt has quit IRC | 21:36 | |
| *** aernhart has quit IRC | 21:36 | |
| *** jroll|dupe has joined #openstack-kolla | 21:44 | |
| *** jroll|dupe has quit IRC | 21:44 | |
| *** jroll|dupe has joined #openstack-kolla | 21:44 | |
| openstackgerrit | Suhail Syed proposed openstack/kolla: Fixed typo in all.yml https://review.openstack.org/336294 | 21:44 |
| *** jroll has quit IRC | 21:45 | |
| *** strigazi has quit IRC | 21:45 | |
| *** mkoderer has quit IRC | 21:45 | |
| *** jroll|dupe is now known as jroll | 21:45 | |
| *** inc0 has joined #openstack-kolla | 21:45 | |
| *** g3ek has quit IRC | 21:46 | |
| inc0 | damn, we lost against Portugal | 21:46 |
| *** haplo37 has quit IRC | 21:46 | |
| *** williamcaban has joined #openstack-kolla | 21:46 | |
| *** jamie_h has quit IRC | 21:47 | |
| *** fragatina has quit IRC | 21:47 | |
| *** haplo37 has joined #openstack-kolla | 21:47 | |
| *** g3ek has joined #openstack-kolla | 21:48 | |
| *** Mr_Broken has quit IRC | 21:49 | |
| *** williamcaban has quit IRC | 21:51 | |
| *** zhiwei has joined #openstack-kolla | 21:51 | |
| *** Mr_Broken has joined #openstack-kolla | 21:51 | |
| *** sdake_ has joined #openstack-kolla | 21:53 | |
| *** sdake_ has quit IRC | 21:53 | |
| *** sdake_ has joined #openstack-kolla | 21:53 | |
| *** zhiwei has quit IRC | 21:55 | |
| *** ravig has joined #openstack-kolla | 21:56 | |
| *** ravig has quit IRC | 21:59 | |
| kfox1111 | very interesting: https://github.com/kubernetes/contrib/pull/1295 | 22:02 |
| kfox1111 | would like to see a poc of mariadb done that way. | 22:03 |
| *** Mr_Broken has quit IRC | 22:04 | |
| *** Mr_Broken has joined #openstack-kolla | 22:05 | |
| *** bootsha has quit IRC | 22:05 | |
| *** banix has quit IRC | 22:14 | |
| *** rhallisey has quit IRC | 22:17 | |
| *** huikang has quit IRC | 22:19 | |
| *** Mr_Broken has quit IRC | 22:19 | |
| *** Mr_Broken has joined #openstack-kolla | 22:21 | |
| *** bootsha has joined #openstack-kolla | 22:25 | |
| *** bootsha has quit IRC | 22:27 | |
| *** banix has joined #openstack-kolla | 22:28 | |
| *** sdake_ has quit IRC | 22:33 | |
| *** vhosakot has quit IRC | 22:34 | |
| *** salv-orl_ has joined #openstack-kolla | 22:34 | |
| *** salv-orlando has quit IRC | 22:35 | |
| *** ravig has joined #openstack-kolla | 22:39 | |
| *** ravig has quit IRC | 22:41 | |
| *** g3ek has quit IRC | 22:43 | |
| openstackgerrit | Christian Berendt proposed openstack/kolla: Integrate neutron-vpnaas-agent into the neutron Ansible role https://review.openstack.org/335502 | 22:44 |
| *** haplo37 has quit IRC | 22:45 | |
| *** williamcaban has joined #openstack-kolla | 22:48 | |
| *** Mr_Broken has quit IRC | 22:49 | |
| *** Mr_Broken has joined #openstack-kolla | 22:50 | |
| *** g3ek has joined #openstack-kolla | 22:50 | |
| *** signed8bit has joined #openstack-kolla | 22:50 | |
| *** haplo37 has joined #openstack-kolla | 22:50 | |
| *** williamcaban has quit IRC | 22:52 | |
| dcwangmit01 | kfox1111: Regarding petsets, we should take a hard look whether they are really needed. They are most definitely needed if you are running a datastore and don't have access to network block storage (ie. Ceph). Otherwise, using petsets is like naming and treating your containers like pets, which is what life was like before Kubernetes came along. A lot of the motivations of petsets... | 22:54 |
| dcwangmit01 | constant local storage, constant network identity, etc can be take care of by persiste volume claims and using kubernetes services. | 22:55 |
| kfox1111 | dcwangmit01: I'm worried without petsets, things like datacenter power outages will cause problems with containers and mariadb galera clusters. | 22:55 |
| kfox1111 | petsets seemed to be designed for dealing with clusters of ha things where members expect to talk to each other without a load balancer. | 22:56 |
| kfox1111 | ie, etcd, rabbit, galera, zookeeper, etc. | 22:56 |
| dcwangmit01 | For example, the POC I posted with single-node mariadb allows the persistent volume to follow the mariadb pod to the next node when the current node (machine) dies: https://review.openstack.org/#/c/335215/ | 22:56 |
| patchbot | dcwangmit01: patch 335215 - kolla-kubernetes - POC (do no merge): Kolla-Kubernetes multi-node per... | 22:56 |
| kfox1111 | if you look at the examples, they use PV's too. | 22:57 |
| kfox1111 | dcwangmit01: yeah, for single container things like the POC, I think it works ok. | 22:57 |
| kfox1111 | but I think it falls apart when count != 1. | 22:57 |
| dcwangmit01 | If you add Galera to the mix, things may change | 22:57 |
| kfox1111 | right. which is the set in petset. :) | 22:58 |
| kfox1111 | I don't think it realy matters until you have a whole set of things. | 22:58 |
| *** banix has quit IRC | 23:00 | |
| dcwangmit01 | Well, not necessarily. Depends on the datastore. For example, wirehead_ and I areu using This plugin for elasticsearch (https://github.com/pires/kubernetes-elasticsearch-cluster). We have many masters, many cleints, and many data nodes. The masters must elect themselves. When we kill nodes, even *all* of the nodes, all the pods scramble to new nodes and their persistent volume claims | 23:00 |
| dcwangmit01 | move along with them. The service discovery happens through kubernetes service registration and querying the kubernetes API. In this case, we don't need a petset, even though the elasticsearch nodes are all of asymmetric types. Using a petset in this case would be detrimental, since you're better off not treating things like pets or restring pods to stickily run on certain nodes | 23:00 |
| dcwangmit01 | But yeah, I think for some legacy, non-cloud aware stuff petsets are needed. Petsets should be used sparingly though | 23:01 |
| kfox1111 | I only see 'emptyDir volume in es-data-rc. | 23:02 |
| kfox1111 | lights out case would see all data deleted I think. | 23:03 |
| kfox1111 | agreed. petsets probably only should be applied to mariadb gallera clusters and maybe rabbit if you want persistant queues. | 23:03 |
| *** banix has joined #openstack-kolla | 23:04 | |
| kfox1111 | basically, the problem I think is if you use a rc, you cant use pvc's as you can't have the same pvc on multiple rc's (usually). | 23:04 |
| dcwangmit01 | Yeah. We've tweaked it to use PVCs. | 23:04 |
| kfox1111 | but petsets you can, as the pvc's are templated. | 23:04 |
| kfox1111 | really? is there a template thing I missed? | 23:04 |
| dcwangmit01 | kfox1111: Okay, so what you said regarding using how an RC cannot map its pods that it creates to various pvc is completely true. That is an unimplemented kubernetes feature that is badly needed. If you have an ES-master RC, with replication factor 5, it would be great to say ensure 5 pods run, and make sure they each mount one of these other 5 volumes any which way (targeted by labels and | 23:06 |
| dcwangmit01 | selectors) | 23:06 |
| dcwangmit01 | ^ So that doesn't exist, so the workaround is to run 5 ES master pods, each with 1 pod within it, targeted to a specific RC. | 23:06 |
| kfox1111 | thats basically how petsets work. | 23:06 |
| dcwangmit01 | This is where jinja-fu comes into play | 23:06 |
| dcwangmit01 | Oh, i read petsets as using local storage on the node. | 23:07 |
| kfox1111 | you have a pvc template as part of the petset, and it allocates a new pvc for each node. | 23:07 |
| kfox1111 | s/node/pod/ | 23:07 |
| *** Mr_Broken has quit IRC | 23:07 | |
| kfox1111 | interesting.. in fact.... | 23:09 |
| kfox1111 | https://github.com/kubernetes/contrib/blob/master/pets/mysql/galera/mysql-galera.yaml | 23:09 |
| dcwangmit01 | Okay, that sounds great. We need that. I'm only reading this, and it's not really clear to me: https://github.com/smarterclayton/kubernetes/blob/961f1f94c35d4979ac83bbad482090cb6c22781c/docs/proposals/petset.md | 23:09 |
| kfox1111 | they are supposed to be working on documentation on it, but haven't gotten it yet. | 23:09 |
| *** ayoung has joined #openstack-kolla | 23:09 | |
| kfox1111 | see the volumeClaimTemplates: at the bottom of that link. | 23:10 |
| kfox1111 | petset as far as I can gather does just a couple of things: | 23:10 |
| kfox1111 | * ensures a container with names petsetname-%index exists somewhere | 23:10 |
| kfox1111 | * allocates a PVC per index, and if the pod has to be deleted/recreated, makes sure the same PVC is used for the matching pod. | 23:11 |
| *** sdake has joined #openstack-kolla | 23:11 | |
| *** sdake has quit IRC | 23:11 | |
| kfox1111 | * ensures a dns entry in skydns corisponding to the petsetname-%index points to that pod always, no matter where it lands or when its recreated. | 23:11 |
| dcwangmit01 | Yeah. I see the VolumeClaimTemplates. That's good. I'm having trouble understanding why they didn't just add that to RCs | 23:11 |
| *** sacharya has quit IRC | 23:11 | |
| kfox1111 | between those 3 features, I think most clustering types of services can work. petset is kind of a bad name for it. | 23:12 |
| kfox1111 | each member can die and be recreated, so its not really a pet. | 23:12 |
| *** ravig has joined #openstack-kolla | 23:12 | |
| kfox1111 | I think they didn't want to add it to RC as rc is intended for loosly coupled things that can scale basically linearlyl. | 23:13 |
| kfox1111 | petsets tend do have containers that want to talk to each other, and can't scale as linearly as say a web server. | 23:13 |
| kfox1111 | otherwise, I think they are very similar. | 23:13 |
| dcwangmit01 | Hmmh. I must be missing something else. RC+Sticky_PVCs. There must be more to petsets ;) | 23:14 |
| dcwangmit01 | Could just be another situation like secrets vs configmaps. They are the same right now except for base64, but in the future secrets will integrate with IAM, etc. | 23:15 |
| dcwangmit01 | Need to know more about petset intentions | 23:15 |
| kfox1111 | yeah. I think they probably share a lot of code. been keeping my eye on them for quite a while. | 23:16 |
| kfox1111 | the docs should land here: https://github.com/kubernetes/kubernetes.github.io/issues/510 | 23:16 |
| kfox1111 | so you might want to keep an eye on that issue. | 23:16 |
| dcwangmit01 | kfox1111: The way I read the proposal, it was that a pod would always run on the same node. But I like the way you are interpreting it better | 23:16 |
| kfox1111 | I think origionally was proposed that way. but its been getting more flexable over time. | 23:17 |
| kfox1111 | the uber thread is here: https://github.com/kubernetes/kubernetes/issues/260 | 23:17 |
| kfox1111 | I sat down for a few hours and read through it once. :/ | 23:17 |
| dcwangmit01 | I see the comment thread, but what I really want to read is the latest draft. Is this not hte latest version: https://github.com/smarterclayton/kubernetes/blob/961f1f94c35d4979ac83bbad482090cb6c22781c/docs/proposals/petset.md | 23:17 |
| kfox1111 | yeah. I read through that one too. | 23:18 |
| kfox1111 | I think it has evolved since then. :/ | 23:18 |
| kfox1111 | though it looks like it still has a varient of persistentVolumeClaim in it. | 23:19 |
| dcwangmit01 | Cool kfox1111 . Thanks for the discussion | 23:20 |
| kfox1111 | np. :) | 23:20 |
| *** berendt has quit IRC | 23:26 | |
| *** signed8bit is now known as signed8bit_Zzz | 23:26 | |
| *** ravig has quit IRC | 23:29 | |
| *** inc0 has quit IRC | 23:40 | |
| *** g3ek has quit IRC | 23:44 | |
| *** haplo37 has quit IRC | 23:45 | |
| *** g3ek has joined #openstack-kolla | 23:50 | |
| *** Mr_Broken has joined #openstack-kolla | 23:50 | |
| *** haplo37 has joined #openstack-kolla | 23:50 | |
| *** banix has quit IRC | 23:55 | |
| *** phuongnh has quit IRC | 23:57 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!