| *** huikang has joined #openstack-kolla | 00:06 | |
| janem | hi all, trying to enable cinder-volume. Do I have to setup vgs cinder-volumes on the docker host machine? | 00:12 |
|---|---|---|
| janem | AIO install | 00:12 |
| janem | 2.0.2 Ubuntu host ubuntu-source images | 00:13 |
| janem | found this: http://docs.openstack.org/developer/kolla/mitaka/cinder-guide.html | 00:15 |
| janem | unclear if it means on host or in cinder-volume container | 00:16 |
| sbezverk_ | janem: on the host, container does not own any vgs | 00:17 |
| janem | sbezverk_: thanks! | 00:17 |
| sbezverk_ | janem: np | 00:18 |
| *** schwicht has quit IRC | 00:23 | |
| MarMat | ping Daviey | 00:29 |
| *** huikang has quit IRC | 00:38 | |
| *** huikang has joined #openstack-kolla | 00:38 | |
| *** huikang_ has joined #openstack-kolla | 00:40 | |
| *** tonanhngo has quit IRC | 00:42 | |
| *** huikang has quit IRC | 00:43 | |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla: Improve VIP existence check https://review.openstack.org/381589 | 00:45 |
| *** eaguilar has quit IRC | 00:48 | |
| *** Pavo has quit IRC | 00:51 | |
| *** dave-mcc_ has joined #openstack-kolla | 00:54 | |
| *** dave-mccowan has quit IRC | 00:55 | |
| *** Pavo has joined #openstack-kolla | 00:56 | |
| *** janem has quit IRC | 00:59 | |
| *** phuongnh has joined #openstack-kolla | 01:00 | |
| *** duonghq has joined #openstack-kolla | 01:00 | |
| *** HyperJohnGraham has joined #openstack-kolla | 01:02 | |
| HyperJohnGraham | Hi All | 01:03 |
| duonghq | hi HyperJohnGraham | 01:04 |
| *** tovin07 has joined #openstack-kolla | 01:08 | |
| *** tovin07 has quit IRC | 01:08 | |
| *** tovin07 has joined #openstack-kolla | 01:09 | |
| *** tovin07 has quit IRC | 01:09 | |
| *** tovin07 has joined #openstack-kolla | 01:10 | |
| *** MarMat has quit IRC | 01:10 | |
| *** tovin07 has quit IRC | 01:11 | |
| HyperJohnGraham | Hi duonghq | 01:11 |
| *** tovin07 has joined #openstack-kolla | 01:12 | |
| *** huikang_ has quit IRC | 01:12 | |
| HyperJohnGraham | sbezverk_: I got some new toys today for the ceph osd's http://liqid.com/liqid_powered/kingston_ep1000.html | 01:12 |
| *** huikang has joined #openstack-kolla | 01:13 | |
| *** huikang has quit IRC | 01:13 | |
| *** huikang has joined #openstack-kolla | 01:13 | |
| *** unicell has quit IRC | 01:25 | |
| *** dwalsh has joined #openstack-kolla | 01:26 | |
| *** dave-mccowan has joined #openstack-kolla | 01:28 | |
| *** dave-mcc_ has quit IRC | 01:29 | |
| *** dwalsh has quit IRC | 01:31 | |
| *** dwalsh has joined #openstack-kolla | 01:44 | |
| *** eaguilar has joined #openstack-kolla | 01:44 | |
| *** MarMat has joined #openstack-kolla | 01:54 | |
| *** tonanhngo has joined #openstack-kolla | 01:55 | |
| *** dave-mccowan has quit IRC | 02:01 | |
| duonghq | coolsvap: did you get this: https://bugs.launchpad.net/kolla/+bug/1630678 | 02:09 |
| openstack | Launchpad bug 1630678 in kolla "keepalived container cannot get MASTER state" [Undecided,New] | 02:09 |
| *** inc0 has quit IRC | 02:09 | |
| *** daneyon has joined #openstack-kolla | 02:10 | |
| *** dwalsh has quit IRC | 02:10 | |
| *** daneyon has quit IRC | 02:15 | |
| *** yuanying has quit IRC | 02:27 | |
| *** haplo37_ has quit IRC | 02:39 | |
| *** haplo37_ has joined #openstack-kolla | 02:41 | |
| *** huikang has quit IRC | 02:50 | |
| *** huikang has joined #openstack-kolla | 02:50 | |
| *** Pavo has quit IRC | 02:51 | |
| *** hieulq has joined #openstack-kolla | 02:53 | |
| *** huikang has quit IRC | 02:55 | |
| *** Pavo has joined #openstack-kolla | 02:55 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 02:55 |
| *** zhubingbing has joined #openstack-kolla | 02:56 | |
| *** jtriley has joined #openstack-kolla | 02:58 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 03:03 |
| *** hieulq has quit IRC | 03:04 | |
| *** huikang has joined #openstack-kolla | 03:08 | |
| *** huikang has quit IRC | 03:12 | |
| *** sdake has joined #openstack-kolla | 03:14 | |
| sdake | yo peeps | 03:14 |
| *** hieulq has joined #openstack-kolla | 03:18 | |
| duonghq | hi sdake | 03:27 |
| sdake | sup dotplus | 03:28 |
| sdake | duonghq | 03:28 |
| duonghq | sdake: some more log about last night (in my tz) bug: https://bugs.launchpad.net/kolla/+bug/1630678 | 03:28 |
| openstack | Launchpad bug 1630678 in kolla "keepalived container cannot get MASTER state" [Undecided,New] | 03:28 |
| duonghq | I'm create totally new VM for recheck | 03:29 |
| sdake | oh vms | 03:31 |
| sdake | yo udidn't mention vms | 03:31 |
| sdake | what is yoru virtual machine manager? | 03:31 |
| duonghq | iirc, I mentioned VBox | 03:31 |
| duonghq | maybe wrong recall after hit my keyboard | 03:32 |
| sdake | i coul dnever get outgoing networkign to work in virtualbox | 03:37 |
| sdake | until i setup a host network | 03:38 |
| sdake | and did a bunch of other magic | 03:38 |
| sdake | which i now don't recall | 03:38 |
| sdake | i had intended to document it | 03:38 |
| sdake | but never did | 03:38 |
| sdake | sounds environmental to me esepcially if your in a vm | 03:38 |
| *** hieulq has quit IRC | 03:38 | |
| sdake | if that hapepned on bare metal, i'd be concerned | 03:38 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 03:39 |
| *** sdake_ has joined #openstack-kolla | 03:45 | |
| *** sdake has quit IRC | 03:47 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 04:01 |
| *** jmccarthy has quit IRC | 04:05 | |
| *** jmccarthy has joined #openstack-kolla | 04:06 | |
| *** haplo37 has quit IRC | 04:07 | |
| *** g3ek has quit IRC | 04:11 | |
| *** yuanying has joined #openstack-kolla | 04:13 | |
| *** g3ek has joined #openstack-kolla | 04:13 | |
| *** haplo37 has joined #openstack-kolla | 04:17 | |
| MarMat | ping sdake_ | 04:19 |
| sdake_ | sup MarMat | 04:19 |
| *** eaguilar has quit IRC | 04:19 | |
| MarMat | sdake the good thing is that cinder + ceph + magnum is working fine for me | 04:20 |
| MarMat | sdake_ not so good things is that there are 2 bug reports opent for magnum deployment by other people | 04:21 |
| sdake_ | does our deploy work? | 04:22 |
| sdake_ | we have to trust that if magnum is doa it wont be when we release | 04:22 |
| *** DanyC has joined #openstack-kolla | 04:22 | |
| MarMat | sdake_ 1st one is that reconfigure/genconfin is not working, fix for this is unfortunately not straightforward and I need some help from Jeffrey4l_ | 04:23 |
| MarMat | sdale_ 2nd one is failure on deployment, I requested further info from reporter https://bugs.launchpad.net/kolla/+bug/1630770 | 04:23 |
| openstack | Launchpad bug 1630770 in kolla "magnum deploy failing" [Undecided,Incomplete] - Assigned to Martin Matyáš (martinx-maty) | 04:23 |
| MarMat | sdake_ ^^^ | 04:24 |
| sdake_ | MarMat that secondbug looks like they are not running as sudo | 04:24 |
| zhubingbing | hello guys | 04:25 |
| sdake_ | Could not create retry file '/usr/share/kolla/ansible/site.retry'. The error was: [Errno 13] Permission denied: '/usr/share/kolla/ansible/site.retry' | 04:25 |
| sdake_ | marmat /usr/share/kolla is supposed to be read only | 04:25 |
| sdake_ | i don't know why a site.retry file is being put in /usr/share/kolla/ansible but that is wrong | 04:26 |
| MarMat | sdake_ well, that's probably only side issue, main concern is "User b43ed8d94a1448e1ad876b2188d636c0 is not valid\" error | 04:26 |
| MarMat | sdake_ during inserting user rules | 04:26 |
| sdake_ | marmat i see | 04:27 |
| sdake_ | marmat we have about 6-7 days depending on how yo ucount to sort em out | 04:27 |
| MarMat | sdake_ but I've never seen that on my systems | 04:27 |
| duonghq | sdake_: for the bug, it used to work before, I will try other hypervisor | 04:28 |
| duonghq | sdake_: also setup host network :) | 04:28 |
| MarMat | sdake_ let's see, I now need cooperation of others, Jeffrey4l_ and Daviey (reporter) | 04:29 |
| sdake_ | MarMat do you have link to the first bug | 04:29 |
| MarMat | https://bugs.launchpad.net/kolla/+bug/1630248 | 04:29 |
| openstack | Launchpad bug 1630248 in kolla "magnum genconfig fails" [High,Triaged] | 04:29 |
| *** britthouser has joined #openstack-kolla | 04:29 | |
| *** britthouser2 has quit IRC | 04:30 | |
| sdake_ | marmat arey ou confirming https://bugs.launchpad.net/kolla/+bug/1630248 is an issue then? | 04:30 |
| openstack | Launchpad bug 1630248 in kolla "magnum genconfig fails" [High,Triaged] | 04:30 |
| MarMat | sdake_ yes, this is confirmed issue | 04:31 |
| sdake_ | marmat why not join the kolla-drivers team to help with bug triage | 04:31 |
| *** unicell has joined #openstack-kolla | 04:32 | |
| MarMat | sdake_ no idea what this means :-) | 04:32 |
| *** unicell has quit IRC | 04:33 | |
| sdake_ | marmat the kolla-drivers team can change bug states | 04:33 |
| sdake_ | like set them to confirmed | 04:33 |
| sdake_ | etc | 04:33 |
| MarMat | sdake this I can do anyway right? I just cannot set Importance. | 04:34 |
| MarMat | sdake_ I already changed status of one of them | 04:35 |
| *** unicell has joined #openstack-kolla | 04:36 | |
| sdake_ | i see | 04:36 |
| sdake_ | i dont know what non-drivers can or can't do :) | 04:36 |
| *** salv-orlando has joined #openstack-kolla | 04:41 | |
| *** yuanying has quit IRC | 04:45 | |
| *** salv-orlando has quit IRC | 04:46 | |
| *** zhubingbing has quit IRC | 04:49 | |
| *** Pavo has quit IRC | 04:51 | |
| *** unicell has quit IRC | 04:51 | |
| *** unicell1 has joined #openstack-kolla | 04:51 | |
| *** yuanying has joined #openstack-kolla | 04:54 | |
| *** Pavo has joined #openstack-kolla | 04:55 | |
| *** bjolo_ has joined #openstack-kolla | 05:06 | |
| *** salv-orlando has joined #openstack-kolla | 05:07 | |
| *** haplo37_ has quit IRC | 05:15 | |
| *** haplo37_ has joined #openstack-kolla | 05:17 | |
| *** david-lyle has joined #openstack-kolla | 05:19 | |
| bjolo | morning | 05:21 |
| *** MarMat has quit IRC | 05:26 | |
| *** DanyC_ has joined #openstack-kolla | 05:28 | |
| *** DanyC_ has quit IRC | 05:29 | |
| *** DanyC has quit IRC | 05:29 | |
| *** jtriley has quit IRC | 05:31 | |
| *** ldeptula has joined #openstack-kolla | 05:38 | |
| *** daneyon has joined #openstack-kolla | 05:46 | |
| *** daneyon has quit IRC | 05:51 | |
| *** egonzalez90 has joined #openstack-kolla | 06:02 | |
| *** yuanying has quit IRC | 06:02 | |
| *** david-lyle_ has joined #openstack-kolla | 06:04 | |
| *** david-lyle has quit IRC | 06:04 | |
| sdake_ | bedtime | 06:05 |
| sdake_ | night folks | 06:05 |
| sdake_ | hey bjolo | 06:05 |
| bjolo | hej | 06:06 |
| bjolo | sup | 06:06 |
| sdake_ | tired | 06:06 |
| sdake_ | 3% batteries | 06:06 |
| sdake_ | time for bed :) | 06:06 |
| bjolo | i hear ya, you are really pushing yourself | 06:08 |
| bjolo | try not to burn yourself out completely | 06:08 |
| *** sdake_ has quit IRC | 06:11 | |
| *** david-lyle_ has quit IRC | 06:12 | |
| *** HyperJohnGraham has quit IRC | 06:18 | |
| *** salv-orlando has quit IRC | 06:29 | |
| *** dupengfei has quit IRC | 06:35 | |
| *** salv-orlando has joined #openstack-kolla | 06:36 | |
| *** tonanhngo has quit IRC | 06:40 | |
| *** salv-orlando has quit IRC | 06:41 | |
| *** athomas has joined #openstack-kolla | 06:44 | |
| *** g3ek has quit IRC | 06:46 | |
| *** tonanhngo has joined #openstack-kolla | 06:48 | |
| *** tonanhngo has quit IRC | 06:49 | |
| *** haplo37 has quit IRC | 06:51 | |
| *** Pavo has quit IRC | 06:51 | |
| *** haplo37 has joined #openstack-kolla | 06:51 | |
| *** g3ek has joined #openstack-kolla | 06:55 | |
| *** Pavo has joined #openstack-kolla | 06:56 | |
| *** unicell has joined #openstack-kolla | 06:57 | |
| *** unicell1 has quit IRC | 06:58 | |
| *** tonanhngo has joined #openstack-kolla | 07:02 | |
| *** tonanhngo has quit IRC | 07:03 | |
| *** b_bezak has joined #openstack-kolla | 07:10 | |
| *** salv-orlando has joined #openstack-kolla | 07:14 | |
| *** yuanying has joined #openstack-kolla | 07:14 | |
| *** jidar has quit IRC | 07:16 | |
| *** jidar_ has joined #openstack-kolla | 07:16 | |
| *** jidar_ is now known as jidar | 07:17 | |
| *** tonanhngo has joined #openstack-kolla | 07:17 | |
| *** tonanhngo has quit IRC | 07:18 | |
| *** Serlex has joined #openstack-kolla | 07:28 | |
| *** bachp has joined #openstack-kolla | 07:31 | |
| *** tonanhngo has joined #openstack-kolla | 07:34 | |
| *** shardy_afk is now known as shardy | 07:34 | |
| *** tonanhngo has quit IRC | 07:35 | |
| *** DanyC has joined #openstack-kolla | 07:36 | |
| *** DanyC has quit IRC | 07:40 | |
| *** bjolo_ has quit IRC | 07:42 | |
| *** dwalsh has joined #openstack-kolla | 07:44 | |
| *** unicell has quit IRC | 07:54 | |
| *** salv-orl_ has joined #openstack-kolla | 07:56 | |
| *** unicell has joined #openstack-kolla | 07:56 | |
| *** salv-orlando has quit IRC | 07:58 | |
| *** haplo37_ has quit IRC | 08:01 | |
| *** tovin07 has quit IRC | 08:03 | |
| *** tovin07 has joined #openstack-kolla | 08:03 | |
| *** haplo37_ has joined #openstack-kolla | 08:03 | |
| *** zhubingbing has joined #openstack-kolla | 08:09 | |
| *** zhurong has joined #openstack-kolla | 08:11 | |
| egonzalez90 | zhubingbing, ping | 08:12 |
| zhubingbing | hi | 08:12 |
| zhubingbing | here | 08:12 |
| *** mgoddard has joined #openstack-kolla | 08:19 | |
| *** DanyC has joined #openstack-kolla | 08:23 | |
| *** daneyon has joined #openstack-kolla | 08:29 | |
| *** mewald has joined #openstack-kolla | 08:31 | |
| *** daneyon has quit IRC | 08:34 | |
| *** Pavo has quit IRC | 08:51 | |
| *** Pavo has joined #openstack-kolla | 08:56 | |
| *** bjolo_ has joined #openstack-kolla | 09:02 | |
| *** tonanhngo has joined #openstack-kolla | 09:04 | |
| *** tonanhngo has quit IRC | 09:07 | |
| *** zhubingbing has quit IRC | 09:09 | |
| *** zhubingbing has joined #openstack-kolla | 09:09 | |
| wznoinsk | Jeffrey4l_: ping | 09:10 |
| duonghq | zhubingbing: ping | 09:18 |
| zhubingbing | here | 09:18 |
| duonghq | long time no see you | 09:18 |
| zhubingbing | 5days | 09:18 |
| zhubingbing | －） | 09:19 |
| duonghq | ya | 09:19 |
| *** ankush has quit IRC | 09:19 | |
| duonghq | zhubingbing: can you give some ref? | 09:19 |
| zhubingbing | Back to hometown without network | 09:19 |
| duonghq | in your bug report and commit | 09:19 |
| zhubingbing | －） | 09:19 |
| zhubingbing | ok | 09:20 |
| duonghq | in your bug, it's permission related but the commit is owner change | 09:20 |
| zhubingbing | hmm | 09:21 |
| zhubingbing | yes | 09:21 |
| zhubingbing | gnocchi user change to root | 09:21 |
| *** tovin07 has quit IRC | 09:21 | |
| duonghq | hmm, so the desc in bug report is no longer valid? | 09:21 |
| *** tovin07 has joined #openstack-kolla | 09:22 | |
| zhubingbing | yes | 09:22 |
| duonghq | please update this, too | 09:23 |
| zhubingbing | ok | 09:24 |
| zhubingbing | have done | 09:24 |
| *** strigazi_AFK is now known as strigazi | 09:25 | |
| *** tonanhngo has joined #openstack-kolla | 09:25 | |
| *** tonanhngo has quit IRC | 09:26 | |
| duonghq | zhubingbing: https://bugs.launchpad.net/kolla/+bug/1586589 -> why do you mark it as invalid? | 09:27 |
| openstack | Launchpad bug 1586589 in kolla "Check 127.0.1.1 in /etc/hosts on Ubuntu" [Low,Invalid] - Assigned to zhubingbing (zhubingbing) | 09:27 |
| zhubingbing | i have done it | 09:29 |
| zhubingbing | wait 5min | 09:29 |
| duonghq | zhubingbing: even you finished it, you should not change the pri and state | 09:31 |
| duonghq | *status | 09:31 |
| duonghq | Gerrit'll changed it to fix released automatically | 09:32 |
| duonghq | the importance should not be changed too, it's bug nature, doesn't depend whether the fix is released or not | 09:32 |
| *** berendt has joined #openstack-kolla | 09:36 | |
| *** mewald1 has joined #openstack-kolla | 09:36 | |
| *** mewald has quit IRC | 09:39 | |
| *** ankush has joined #openstack-kolla | 09:41 | |
| zhubingbing | duonghq thank you | 09:41 |
| zhubingbing | i know | 09:41 |
| duonghq | wznoinsk: aroud? | 09:44 |
| wznoinsk | duonghq: hi | 09:44 |
| duonghq | for the bug: https://bugs.launchpad.net/kolla/+bug/1613294 | 09:45 |
| openstack | Launchpad bug 1613294 in kolla "sudo ironic-rootwrap /etc/ironic/rootwrap.conf iscsiadm -m discovery -t st -p <IP> hangs forever" [High,In progress] - Assigned to Waldemar Znoinski (wznoinsk) | 09:45 |
| duonghq | as I understand, it's already addressed in https://review.openstack.org/#/c/379106/4 | 09:45 |
| zhubingbing | hi | 09:46 |
| zhubingbing | https://bugs.launchpad.net/kolla/+bug/1586589 | 09:46 |
| openstack | Launchpad bug 1586589 in kolla "Check 127.0.1.1 in /etc/hosts on Ubuntu" [High,In progress] - Assigned to zhubingbing (zhubingbing) | 09:46 |
| zhubingbing | I think this bug has been solved. | 09:46 |
| zhubingbing | https://github.com/openstack/kolla/blob/master/ansible/roles/prechecks/tasks/port_checks.yml#L442 | 09:47 |
| zhubingbing | duonhq | 09:47 |
| duonghq | zhubingbing: let me check when the line is commit | 09:47 |
| duonghq | moment | 09:47 |
| zhubingbing | ok | 09:47 |
| zhubingbing | I think this bug has been solved. | 09:48 |
| duonghq | zhubingbing: so I think the 127.0.1.1 check should be removed? but worth another bug | 09:49 |
| duonghq | but I'm not really sure the port check is neccesarry or not | 09:50 |
| zhubingbing | i thinks removed | 09:50 |
| duonghq | did you test deploy with the check is removed? | 09:51 |
| *** salv-orl_ has quit IRC | 09:51 | |
| duonghq | make sure the rabbitmq is up and run smoothly | 09:51 |
| zhubingbing | I have been tested, hostname rabbitmq parsing problem | 09:51 |
| duonghq | multi-node tested? | 09:51 |
| zhubingbing | all－in－one | 09:51 |
| duonghq | 1 node? | 09:52 |
| zhubingbing | yeah | 09:52 |
| zhubingbing | multi－node no problmes | 09:52 |
| zhubingbing | i have tested it 3days ago | 09:52 |
| duonghq | I think is not sufficient, although it's should be ok as the logic is correct | 09:52 |
| duonghq | but, please push a ps | 09:53 |
| duonghq | can you do this, zhubingbing? | 09:53 |
| duonghq | remember tag the bug | 09:53 |
| zhubingbing | What I need to do？ | 09:53 |
| duonghq | Create a ps, which remove the 127.0.1.1 check | 09:54 |
| *** egonzalez90 has quit IRC | 09:54 | |
| duonghq | tag with this bug ID | 09:54 |
| duonghq | and put a ref to paul's ps in the commit msg | 09:54 |
| duonghq | If it get its way to master, so the bug is automatically closed. | 09:54 |
| zhubingbing | ok | 09:54 |
| *** egonzalez90 has joined #openstack-kolla | 09:54 | |
| duonghq | I have some trouble with the VBox VM, it's cannot deploy kolla | 09:55 |
| duonghq | so I cannot verify the solution quick | 09:55 |
| duonghq | sorry | 09:55 |
| *** awiddersheim has quit IRC | 09:55 | |
| duonghq | zhubingbing: you test on physical machine or VM? | 09:55 |
| duonghq | (just for information) | 09:56 |
| *** msimonin has joined #openstack-kolla | 09:56 | |
| zhubingbing | vm | 09:56 |
| duonghq | VBox or kvm? | 09:56 |
| zhubingbing | kvm | 09:56 |
| duonghq | fyi, the Paul commit id is 6c238ca3, | 09:56 |
| duonghq | I should put correspond change id to your commit for reference | 09:57 |
| zhubingbing | How did you find it | 09:57 |
| duonghq | git blame do the magic | 09:57 |
| zhubingbing | －）wow | 09:57 |
| zhubingbing | https://bugs.launchpad.net/kolla/+bug/1586589 | 09:58 |
| openstack | Launchpad bug 1586589 in kolla "Check 127.0.1.1 in /etc/hosts on Ubuntu" [High,In progress] - Assigned to zhubingbing (zhubingbing) | 09:58 |
| zhubingbing | This bug you to close it？ | 09:58 |
| duonghq | need a merged ps for closing this bug. | 09:59 |
| duonghq | yeah, this bug | 09:59 |
| duonghq | absolutely | 09:59 |
| zhubingbing | https://github.com/openstack/kolla/commit/6c238ca3bfc082813252098735e3fd6062ad952e | 10:01 |
| zhubingbing | commit ps | 10:01 |
| duonghq | wznoinsk: does the fix for https://bugs.launchpad.net/kolla/+bug/1613294 is backported to Mitaka? | 10:01 |
| openstack | Launchpad bug 1613294 in kolla mitaka "sudo ironic-rootwrap /etc/ironic/rootwrap.conf iscsiadm -m discovery -t st -p <IP> hangs forever" [High,Confirmed] - Assigned to Waldemar Znoinski (wznoinsk) | 10:01 |
| duonghq | thanks zhubingbing | 10:02 |
| zhubingbing | np | 10:02 |
| zhubingbing | you done it | 10:02 |
| zhubingbing | close it | 10:02 |
| duonghq | zhubingbing: the bug is resolved only when the check is remove from the playbook | 10:03 |
| duonghq | or I'll make the ps? | 10:03 |
| zhubingbing | I think it ok | 10:05 |
| duonghq | ok, so I'll push the fix | 10:05 |
| duonghq | 5min, get task jam, | 10:06 |
| zhubingbing | thank you | 10:06 |
| zhubingbing | I want to integrate cloudkitty and gnocchi | 10:06 |
| zhubingbing | have you use cloud kitty？ | 10:07 |
| duonghq | not ready, iirc there is cloudkitty role? | 10:07 |
| *** Serlex has quit IRC | 10:11 | |
| *** awiddersheim has joined #openstack-kolla | 10:12 | |
| wznoinsk | duonghq: sorry, in a meeting atm, fix for https://bugs.launchpad.net/kolla/+bug/1613294 was the sudoers which is now in https://review.openstack.org/#/c/379106/4 | 10:12 |
| openstack | Launchpad bug 1613294 in kolla mitaka "sudo ironic-rootwrap /etc/ironic/rootwrap.conf iscsiadm -m discovery -t st -p <IP> hangs forever" [High,Confirmed] - Assigned to Waldemar Znoinski (wznoinsk) | 10:12 |
| duonghq | thank wznoinsk, marked it as fix released | 10:13 |
| wznoinsk | duonghq: check with Jeffrey4l_ about backporting | 10:14 |
| duonghq | wznoinsk: will do, seem that no backport in queue yet | 10:15 |
| *** tovin07 has quit IRC | 10:18 | |
| *** tonanhngo has joined #openstack-kolla | 10:19 | |
| *** tonanhngo has quit IRC | 10:20 | |
| *** mnasiadka has joined #openstack-kolla | 10:22 | |
| *** salv-orlando has joined #openstack-kolla | 10:23 | |
| *** zhubingbing has quit IRC | 10:29 | |
| *** duonghq has left #openstack-kolla | 10:30 | |
| *** DanyC has quit IRC | 10:34 | |
| *** egonzalez90 has quit IRC | 10:37 | |
| b_bezak | hi | 10:41 |
| b_bezak | how is "kolla-ansible upgrade" tested? ;) | 10:42 |
| b_bezak | I am preparing to upgrade from 2.0.2 to 2.0.3dev (with recent massive ubuntu-cloud archive openstack components upgrades, btw) | 10:42 |
| b_bezak | pushed images to local registry with new tag | 10:43 |
| b_bezak | however I am little bit worried how nova_livbvirt with running vms will behave ;) | 10:43 |
| b_bezak | and ceph | 10:43 |
| b_bezak | well, maybe ceph not so much, compared to nova_libvirt, thougj | 10:44 |
| b_bezak | though | 10:45 |
| *** zhubingbing has joined #openstack-kolla | 10:49 | |
| *** bachp has quit IRC | 10:50 | |
| *** Pavo has quit IRC | 10:51 | |
| *** Pavo has joined #openstack-kolla | 10:56 | |
| *** Serlex has joined #openstack-kolla | 10:57 | |
| *** shardy is now known as shardy_lunch | 11:01 | |
| *** bachp has joined #openstack-kolla | 11:05 | |
| openstackgerrit | Christian Berendt proposed openstack/kolla: Install python-mysqldb for Ubuntu in openstack-base container https://review.openstack.org/382398 | 11:12 |
| *** daneyon has joined #openstack-kolla | 11:12 | |
| *** ankush has quit IRC | 11:14 | |
| *** bachp has quit IRC | 11:15 | |
| *** dwalsh has quit IRC | 11:16 | |
| *** daneyon has quit IRC | 11:16 | |
| *** rhallisey has joined #openstack-kolla | 11:16 | |
| *** qwang has quit IRC | 11:17 | |
| *** qwang has joined #openstack-kolla | 11:17 | |
| *** msimonin has quit IRC | 11:20 | |
| *** bachp has joined #openstack-kolla | 11:20 | |
| *** msimonin has joined #openstack-kolla | 11:29 | |
| *** ankush has joined #openstack-kolla | 11:30 | |
| *** shardy_lunch is now known as shardy | 11:31 | |
| kfox1111 | sbezverk_: rhallisey: http://logs.openstack.org/41/381041/58/experimental/gate-kolla-kubernetes-ansible-workflow-ceph-nv/2b9f788/console.html | 11:34 |
| kfox1111 | first time all the way through. :) | 11:34 |
| *** eaguilar has joined #openstack-kolla | 11:35 | |
| kfox1111 | vm's launched, cinder attached, fips. :) | 11:35 |
| kfox1111 | all out of ceph. | 11:35 |
| rhallisey | nice! | 11:35 |
| kfox1111 | going to add a few more checks, some polish, and make sure its stable. bit I think its really close now. :) | 11:36 |
| berendt | great news :) | 11:36 |
| *** phuongnh has quit IRC | 11:37 | |
| sbezverk_ | kfox1111: super cool!! | 11:38 |
| *** egonzalez90 has joined #openstack-kolla | 11:43 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 11:44 |
| openstackgerrit | caoyuan proposed openstack/kolla: test please do not merge https://review.openstack.org/382874 | 11:49 |
| *** eaguilar has quit IRC | 11:51 | |
| *** hyakuhei- has quit IRC | 11:52 | |
| *** hyakuhei- has joined #openstack-kolla | 11:52 | |
| *** hyakuhei- has quit IRC | 11:52 | |
| *** hyakuhei- has joined #openstack-kolla | 11:52 | |
| *** hyakuhei- is now known as hyakuhei | 11:52 | |
| openstackgerrit | Mauricio Lima proposed openstack/kolla: [WIP] Add HNAS as Manila backend https://review.openstack.org/382054 | 11:54 |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla: Unmount Ceph OSD disks as part of destroy https://review.openstack.org/380028 | 11:59 |
| mliima | morning guys | 11:59 |
| jmccarthy | If I get this error 'The ipaddr filter requires python-netaddr be installed on the ansible controller' - what is the "ansible controller" ? | 12:04 |
| berendt | jmccarthy the host where you run kolla-ansible | 12:05 |
| jmccarthy | berendt, ok not the nodes I'm trying to deploy to, or in the toolbox container or something like this, you mean the 'operator' or 'master' node ? | 12:05 |
| berendt | operator node, master node, ansible controller, ... | 12:06 |
| berendt | it is all the same | 12:06 |
| jmccarthy | Ok cool, never heard of 'ansible controller' before lol | 12:07 |
| berendt | in ansible/roles/prechecks/tasks/port_checks.yml we use this specific filter | 12:07 |
| *** zhurong has quit IRC | 12:07 | |
| jmccarthy | Yep, that's the one I have an issue in my env, will dig some more | 12:07 |
| berendt | it should be sufficient to simply install python-netaddr | 12:07 |
| *** zhurong has joined #openstack-kolla | 12:08 | |
| berendt | i think ansible controller is a bad word for the master node.. for me a controller is something active | 12:08 |
| jmccarthy | and 'lo and behold - it is :) | 12:08 |
| jmccarthy | Just wanted to be clear where I was supposed to be installing it ;) | 12:08 |
| *** mgoddard_ has joined #openstack-kolla | 12:08 | |
| berendt | looks like we use a different word: "deployment host" | 12:09 |
| pbourke | whats meant to drag that in | 12:10 |
| pbourke | we dont document to install it anywhere | 12:10 |
| openstackgerrit | Christian Berendt proposed openstack/kolla: Add python-netaddr as required dependency on the deployment host https://review.openstack.org/382892 | 12:11 |
| *** mgoddard has quit IRC | 12:11 | |
| berendt | pbourke here you go | 12:12 |
| pbourke | should we add it to requirements.txt? | 12:13 |
| berendt | it is a requirement of kolla-ansible | 12:14 |
| berendt | to we track those requirements inside requirements.txt? | 12:14 |
| berendt | to = do | 12:14 |
| berendt | jinja2 is also listed in requirements.txt.. but it is also used by kolla-build | 12:14 |
| *** mgoddard_ has quit IRC | 12:15 | |
| *** Jeffrey4l_ has quit IRC | 12:20 | |
| *** schwicht has joined #openstack-kolla | 12:20 | |
| *** Jeffrey4l_ has joined #openstack-kolla | 12:21 | |
| *** mgoddard has joined #openstack-kolla | 12:22 | |
| *** salv-orlando has quit IRC | 12:24 | |
| *** lamt has quit IRC | 12:25 | |
| *** srwilkers has joined #openstack-kolla | 12:33 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 12:34 |
| *** fguillot has joined #openstack-kolla | 12:38 | |
| mnasiadka | guys, anybody deployed a fresh master on ubuntu xenial with python 3.5? | 12:40 |
| mnasiadka | http://paste.openstack.org/show/584665/ | 12:40 |
| mnasiadka | kolla-genpwd fails... | 12:40 |
| openstackgerrit | zhubingbing proposed openstack/kolla: integrate gnocchi with cloudkitty Closes-Bug: #1630947 https://review.openstack.org/382916 | 12:43 |
| openstack | bug 1630947 in kolla "cloudkitty do not support gnocchi collector" [High,In progress] https://launchpad.net/bugs/1630947 - Assigned to zhubingbing (zhubingbing) | 12:43 |
| *** rodrigo_pereira has joined #openstack-kolla | 12:44 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 12:47 |
| *** srwilkers has quit IRC | 12:47 | |
| wznoinsk | Jeffrey4l_: I see your change https://review.openstack.org/#/c/379106/4 assuming iscsid running somewhere, have you tested it with pxe_ipmitool maybe? | 12:49 |
| *** Pavo has quit IRC | 12:51 | |
| *** srwilkers has joined #openstack-kolla | 12:51 | |
| *** Pavo has joined #openstack-kolla | 12:55 | |
| openstackgerrit | Merged openstack/kolla-kubernetes: literal block in quickstart are messed up so, blank lines added in literal block and some Inline Markup added https://review.openstack.org/382454 | 12:57 |
| *** ankush has quit IRC | 13:01 | |
| *** jheroux has joined #openstack-kolla | 13:02 | |
| *** jtriley has joined #openstack-kolla | 13:03 | |
| *** dave-mccowan has joined #openstack-kolla | 13:04 | |
| *** ayoung has joined #openstack-kolla | 13:04 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: integrate gnocchi with cloudkitty https://review.openstack.org/382916 | 13:04 |
| *** inc0 has joined #openstack-kolla | 13:04 | |
| inc0 | good morning | 13:05 |
| zhubingbing | morning | 13:05 |
| *** ankush has joined #openstack-kolla | 13:08 | |
| inc0 | Jeffrey4l_, ping | 13:09 |
| openstackgerrit | Mauricio Lima proposed openstack/kolla: Remove default values from conf file https://review.openstack.org/382936 | 13:15 |
| *** huikang has joined #openstack-kolla | 13:19 | |
| *** msimonin has quit IRC | 13:23 | |
| *** msimonin has joined #openstack-kolla | 13:24 | |
| *** mkoderer has joined #openstack-kolla | 13:27 | |
| *** haplo37_ has quit IRC | 13:28 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: integrate gnocchi with cloudkitty https://review.openstack.org/382916 | 13:29 |
| *** haplo37_ has joined #openstack-kolla | 13:30 | |
| *** jtriley has quit IRC | 13:31 | |
| *** jtriley has joined #openstack-kolla | 13:32 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: integrate gnocchi with cloudkitty https://review.openstack.org/382916 | 13:33 |
| sbezverk_ | morning, is newton out? | 13:35 |
| *** huikang has quit IRC | 13:35 | |
| inc0 | sbezverk_, 12 oct | 13:35 |
| mnasiadka | ok, uninstalling abandoned pycrypto and installing pycryptodome did the trick | 13:36 |
| *** huikang has joined #openstack-kolla | 13:36 | |
| sbezverk_ | inc0: I guess cinder folks are early to deliver it ;-) | 13:36 |
| inc0 | nah, we're late:) | 13:36 |
| sbezverk_ | inc0: :-) got it | 13:37 |
| inc0 | we release later than others | 13:37 |
| inc0 | because we depend on them:) | 13:37 |
| *** jtriley has quit IRC | 13:37 | |
| sbezverk_ | inc0: I like this approach, give a bit of extra time ;-) | 13:38 |
| inc0 | well, we need to bind to stable releases and then test ourselves out | 13:39 |
| *** huikang has quit IRC | 13:40 | |
| *** huikang has joined #openstack-kolla | 13:45 | |
| *** msimonin has quit IRC | 13:49 | |
| *** msimonin has joined #openstack-kolla | 13:51 | |
| *** daneyon has joined #openstack-kolla | 13:54 | |
| *** ankush has quit IRC | 13:54 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 13:55 |
| *** msimonin has quit IRC | 13:55 | |
| *** MarMat has joined #openstack-kolla | 13:56 | |
| *** tonanhngo has joined #openstack-kolla | 13:56 | |
| Jeffrey4l_ | inc0, pong | 13:56 |
| *** MarMat has quit IRC | 13:57 | |
| *** MarMat has joined #openstack-kolla | 13:57 | |
| inc0 | Jeffrey4l_, do you know if Ironic is working well? | 13:57 |
| Jeffrey4l_ | inc0, it is. except the iscsid parted. | 13:58 |
| Jeffrey4l_ | wznoinsk, still around? hmm, yes, iscsid should be started. and i forget to enable iscsid in the code when ironic is enabled. | 13:58 |
| *** daneyon has quit IRC | 13:58 | |
| inc0 | good to know:) I'll need to deploy multi region with ironic | 13:59 |
| inc0 | brb | 13:59 |
| Jeffrey4l_ | inc0, cool i tested the ironic with my fix. it works fine. inc0 you should enable iscsid service too. this is not enabled automatically, now.. | 13:59 |
| *** sdake has joined #openstack-kolla | 13:59 | |
| sdake | bjolo around? | 14:00 |
| wznoinsk | Jeffrey4l_: you mean like: enable_iscsid: "{{ enable_cinder_backend_iscsi | bool or enable_cinder_backend_lvm | bool }}" | 14:01 |
| bjolo | yup | 14:01 |
| Jeffrey4l_ | wznoinsk, yes | 14:01 |
| wznoinsk | Jeffrey4l_: you could help me with this https://review.openstack.org/#/c/381942/ | 14:02 |
| bjolo | sdake, sup | 14:02 |
| sdake | bjolo re burnout last night, i am 42 years old - know how to manage myself | 14:03 |
| Jeffrey4l_ | wznoinsk, ok. np | 14:03 |
| *** pbourke has quit IRC | 14:03 | |
| *** pbourke has joined #openstack-kolla | 14:04 | |
| sdake | bjolo re pushing myself -that is what I do all the time :) | 14:04 |
| bjolo | :) | 14:04 |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla-kubernetes: Workflow Gate Testing https://review.openstack.org/382021 | 14:04 |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla-kubernetes: Ansible support for Mariadb https://review.openstack.org/334255 | 14:04 |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla-kubernetes: Ansible support for Rabbitmq https://review.openstack.org/380796 | 14:04 |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla-kubernetes: Ansible workflow for Keystone https://review.openstack.org/380797 | 14:04 |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla-kubernetes: Ansible workflow for Glance https://review.openstack.org/380801 | 14:04 |
| *** ankush has joined #openstack-kolla | 14:04 | |
| Jeffrey4l_ | wznoinsk, done | 14:09 |
| *** zhenguo has joined #openstack-kolla | 14:11 | |
| *** sdake has quit IRC | 14:15 | |
| *** egonzalez90 has quit IRC | 14:15 | |
| *** bjolo_ has quit IRC | 14:18 | |
| *** athomas has quit IRC | 14:19 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 14:19 |
| sbezverk_ | inc0: rhallisey: kfox1111: please check your mail box :-) | 14:23 |
| *** neilus has joined #openstack-kolla | 14:23 | |
| *** athomas has joined #openstack-kolla | 14:25 | |
| inc0 | sbezverk_, 2 things | 14:25 |
| inc0 | 1. when you create mariadb volume in ceph | 14:26 |
| inc0 | try to add --shared flag | 14:26 |
| inc0 | and see if it fixes ceph issue | 14:26 |
| inc0 | 2. 11 min is horribly long, we need to figure out why it's so long | 14:26 |
| inc0 | I'm still struggling to deploy kolla-k8s with ceph at all | 14:26 |
| sbezverk_ | inc0: I was observing this process and it seems node failure detection was farly quick, but the process of terminating impacted pod took very long time | 14:27 |
| sbezverk_ | node was in NotReady state but pods which were hosted by this server were in running state for at least 6-7 minutes | 14:28 |
| inc0 | but if termination will be induced by node fail | 14:28 |
| inc0 | like actual fail | 14:29 |
| sbezverk_ | inc0: it supposed to be by default | 14:29 |
| *** mgoddard_ has joined #openstack-kolla | 14:29 | |
| inc0 | I wonder how long that would take | 14:29 |
| inc0 | it makes sense to try to gracefully terminate pod before migration | 14:29 |
| *** mgoddard has quit IRC | 14:31 | |
| sbezverk_ | inc0: that is for sure, but when node looses power, no time for graceful termination ;-) | 14:31 |
| inc0 | what I'm asking is if we simulate power loss, and k8s won't care for graceful termination, will it be faster | 14:32 |
| inc0 | 11min is WAY too long, we need to figure out how to bring that down for demo | 14:32 |
| sbezverk_ | inc0: that is what I did, powered off the node | 14:32 |
| *** rhallisey has quit IRC | 14:36 | |
| *** neilus has quit IRC | 14:38 | |
| *** bmace has quit IRC | 14:39 | |
| *** bmace has joined #openstack-kolla | 14:39 | |
| *** jtriley has joined #openstack-kolla | 14:43 | |
| *** b_bezak has quit IRC | 14:45 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 14:47 |
| *** neilus has joined #openstack-kolla | 14:48 | |
| *** lrensing has joined #openstack-kolla | 14:50 | |
| *** huikang has quit IRC | 14:50 | |
| *** Pavo has quit IRC | 14:51 | |
| *** huikang has joined #openstack-kolla | 14:51 | |
| *** zhurong has quit IRC | 14:52 | |
| *** HyperJohnGraham has joined #openstack-kolla | 14:52 | |
| *** neilus has quit IRC | 14:52 | |
| *** akwasnie has quit IRC | 14:53 | |
| *** kong has quit IRC | 14:53 | |
| *** SaMnCo has quit IRC | 14:54 | |
| *** amaged__ has quit IRC | 14:55 | |
| *** DuncanT has quit IRC | 14:55 | |
| *** mkoderer has quit IRC | 14:55 | |
| *** huikang has quit IRC | 14:55 | |
| *** macsz has quit IRC | 14:55 | |
| *** mgkwill has quit IRC | 14:55 | |
| *** qwang has quit IRC | 14:55 | |
| *** peterjenkins_ has quit IRC | 14:55 | |
| *** ediardo has quit IRC | 14:55 | |
| *** smekel_ has quit IRC | 14:55 | |
| *** coolsvap has quit IRC | 14:55 | |
| *** Pavo has joined #openstack-kolla | 14:55 | |
| *** egonzalez90 has joined #openstack-kolla | 14:56 | |
| *** tmcpeak has joined #openstack-kolla | 14:56 | |
| *** capnoday has joined #openstack-kolla | 14:57 | |
| sbezverk_ | rhallisey: ping | 14:57 |
| capnoday | sup guys | 14:57 |
| tmcpeak | lol | 14:57 |
| tmcpeak | o/ | 14:58 |
| kfox1111 | hi | 14:58 |
| tmcpeak | security review time? | 14:58 |
| tmcpeak | inc0: | 14:58 |
| *** haplo37 has quit IRC | 14:58 | |
| *** g3ek has quit IRC | 14:59 | |
| inc0 | yup, thanks tmcpeak | 14:59 |
| *** salv-orlando has joined #openstack-kolla | 14:59 | |
| tmcpeak | want to hop on google hangouts? | 14:59 |
| inc0 | Jeffrey4l_, pbourke | 14:59 |
| inc0 | yeah, sure | 14:59 |
| inc0 | I guess I forgot to send a link, hold on | 15:00 |
| *** egonzalez90 has quit IRC | 15:00 | |
| inc0 | tmcpeak, Jeffrey4l_ pbourke https://hangouts.google.com/call/quzp3kedrbdtfcbhyts6753ihee | 15:00 |
| tmcpeak | capnoday: ^ | 15:00 |
| *** haplo37 has joined #openstack-kolla | 15:01 | |
| Jeffrey4l_ | roger. | 15:01 |
| pbourke | hmm I dont have a mic handy | 15:01 |
| pbourke | sorry | 15:01 |
| *** g3ek has joined #openstack-kolla | 15:01 | |
| inc0 | https://etherpad.openstack.org/p/kolla-threat-analysis pbourke , join if you can,even if just to listen | 15:03 |
| capnoday | #link https://review.openstack.org/#/c/357978/ | 15:05 |
| *** diogogmt has joined #openstack-kolla | 15:06 | |
| *** huikang has joined #openstack-kolla | 15:07 | |
| sbezverk_ | inc0: I got some ideas from kubernetes slack community, will implement it and retry my test | 15:07 |
| inc0 | ok | 15:08 |
| otavio | We have done some fixes for mitaka branch which easy the development using Vagrent. Most are backports from master. How we should submit those? | 15:09 |
| *** peterjenkins_ has joined #openstack-kolla | 15:11 | |
| *** mnasiadka has quit IRC | 15:15 | |
| *** msimonin has joined #openstack-kolla | 15:15 | |
| *** msimonin has quit IRC | 15:16 | |
| *** mkoderer has joined #openstack-kolla | 15:21 | |
| *** dwalsh has joined #openstack-kolla | 15:21 | |
| *** vhosakot has joined #openstack-kolla | 15:21 | |
| *** akwasnie has joined #openstack-kolla | 15:22 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 15:24 |
| *** amaged__ has joined #openstack-kolla | 15:25 | |
| *** SaMnCo has joined #openstack-kolla | 15:25 | |
| *** mgkwill has joined #openstack-kolla | 15:27 | |
| *** kong has joined #openstack-kolla | 15:34 | |
| *** smekel_ has joined #openstack-kolla | 15:36 | |
| *** DuncanT has joined #openstack-kolla | 15:36 | |
| *** ccesario has quit IRC | 15:36 | |
| *** janem has joined #openstack-kolla | 15:39 | |
| janem | hi all | 15:39 |
| janem | I get an error with deploying on 2.0.2 AIO Ubuntu host. | 15:40 |
| janem | starting kolla-toolbox container | 15:40 |
| janem | TypeError("'NoneType' object is not iterable",) | 15:40 |
| *** macsz has joined #openstack-kolla | 15:41 | |
| openstackgerrit | Mauricio Lima proposed openstack/kolla: Remove unneeded blank lines https://review.openstack.org/383045 | 15:42 |
| kfox1111 | sbezverk_: what ideas? | 15:42 |
| *** ediardo has joined #openstack-kolla | 15:43 | |
| *** tonanhngo has quit IRC | 15:46 | |
| sbezverk_ | kfox1111: there are a couple of timers to play | 15:46 |
| *** tonanhngo has joined #openstack-kolla | 15:46 | |
| kfox1111 | timers? | 15:46 |
| *** coolsvap has joined #openstack-kolla | 15:47 | |
| *** qwang has joined #openstack-kolla | 15:48 | |
| *** msimonin has joined #openstack-kolla | 15:51 | |
| *** srwilkers has quit IRC | 15:53 | |
| *** diogogmt_ has joined #openstack-kolla | 15:53 | |
| *** diogogmt has quit IRC | 15:53 | |
| *** diogogmt_ is now known as diogogmt | 15:53 | |
| *** msimonin1 has joined #openstack-kolla | 15:53 | |
| *** berendt has quit IRC | 15:55 | |
| *** dwalsh has quit IRC | 15:55 | |
| *** msimonin1 has quit IRC | 15:55 | |
| *** msimonin has quit IRC | 15:55 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 15:56 |
| *** haplo37_ has quit IRC | 15:57 | |
| *** nihilifer has quit IRC | 15:58 | |
| *** nihilifer has joined #openstack-kolla | 15:58 | |
| *** haplo37_ has joined #openstack-kolla | 15:59 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: fix keystone log repeated collection in heka https://review.openstack.org/383066 | 16:00 |
| *** david-lyle has joined #openstack-kolla | 16:01 | |
| *** jheroux has quit IRC | 16:02 | |
| *** neilus has joined #openstack-kolla | 16:02 | |
| *** huikang has quit IRC | 16:02 | |
| *** huikang has joined #openstack-kolla | 16:03 | |
| *** jheroux has joined #openstack-kolla | 16:06 | |
| *** huikang has quit IRC | 16:07 | |
| *** huikang has joined #openstack-kolla | 16:07 | |
| *** rhallisey has joined #openstack-kolla | 16:07 | |
| inc0 | pbourke, Jeffrey4l_ can you help me fill this diagram during next couple of days | 16:11 |
| inc0 | ? | 16:11 |
| pbourke | inc0: sure | 16:11 |
| Jeffrey4l_ | inc0, np | 16:11 |
| *** neilus__ has joined #openstack-kolla | 16:12 | |
| inc0 | thanks, do you understand process? whatever I did made sense?:) | 16:12 |
| inc0 | also thanks a lot tmcpeak capnoday | 16:12 |
| tmcpeak | inc0: thanks! great working with you all on this | 16:13 |
| *** neilus has quit IRC | 16:13 | |
| Jeffrey4l_ | inc0, should we draw all the containers and mounts? or part of them? | 16:13 |
| *** capnoday has left #openstack-kolla | 16:13 | |
| *** tmcpeak has left #openstack-kolla | 16:13 | |
| inc0 | Jeffrey4l_, let's start with default config | 16:13 |
| Jeffrey4l_ | ok. | 16:13 |
| inc0 | default services, maybe plus ceph+cinder | 16:13 |
| inc0 | I expect this to be deployment arch that will interest most of people | 16:14 |
| inc0 | after that we will continously work on it, but let's get defaults done first | 16:14 |
| Jeffrey4l_ | Got it. | 16:14 |
| Jeffrey4l_ | yes | 16:14 |
| inc0 | in fact, after we're done we should make this required part of any patchset introducing new service:) | 16:15 |
| openstackgerrit | zhubingbing proposed openstack/kolla: fix can't collect horizon log in heka https://review.openstack.org/383080 | 16:15 |
| inc0 | (figure out how to make gerrit review out of it;)) | 16:15 |
| inc0 | but that's for later | 16:16 |
| inc0 | bbiaf | 16:16 |
| Jeffrey4l_ | gerrit for draw.io is hard i think. may be graphiv is a better tool | 16:16 |
| *** strigazi is now known as strigazi_AFK | 16:17 | |
| *** msimonin has joined #openstack-kolla | 16:18 | |
| *** msimonin has quit IRC | 16:19 | |
| *** rstarmer has joined #openstack-kolla | 16:23 | |
| rstarmer | has anyone used the netapp via NFS? | 16:25 |
| rstarmer | for cinder/glance | 16:25 |
| *** msimonin has joined #openstack-kolla | 16:25 | |
| *** HyperJohnGraham has quit IRC | 16:27 | |
| *** zhubingbing has quit IRC | 16:27 | |
| *** huikang has quit IRC | 16:29 | |
| *** huikang has joined #openstack-kolla | 16:30 | |
| sbezverk_ | kfox1111: there are three timers I changed in kube-controller-manager which reduced running phase of a pod on a notready node considerably, not so good news is Terminating phase did not change. It seems for non petset containers it is ok as deployment or rc starts fresh copy and does not care about terminating old instance | 16:30 |
| *** zhubingbing has joined #openstack-kolla | 16:31 | |
| pbourke | inc0: Jeffrey4l_: you guys test multinode right | 16:31 |
| sbezverk_ | but for pet-set new instance does not get created until old is completely gone. and in our case these are most important services for us mariadb/rabbitmq and nova api | 16:31 |
| *** papacz has quit IRC | 16:31 | |
| inc0 | pbourke, yeah | 16:31 |
| sbezverk_ | kfox1111: but the worst part is second failover crashed my server :-( | 16:31 |
| pbourke | inc0: have you ever seen the setup module just randomly error out in the middle of a playbook run | 16:31 |
| pbourke | inc0: just dumps all facts and basically says "module failure" | 16:32 |
| inc0 | nope.. | 16:32 |
| pbourke | inc0: I see it constantly to the point where it's getting hard to test anything :( | 16:32 |
| zhubingbing | what‘mater for ci gate？ | 16:32 |
| zhubingbing | http://logs.openstack.org/66/383066/1/check/gate-kolla-dsvm-build-centos-binary-centos-7-nv/1153426/console.html#_2016-10-06_16_06_24_017233 | 16:32 |
| pbourke | must be something with my network but can't nail it down | 16:32 |
| inc0 | pbourke, mind pasting it somewhere? | 16:32 |
| Jeffrey4l_ | pbourke, no. need the inventory file and ansible logs | 16:33 |
| *** zhenguo has quit IRC | 16:33 | |
| pbourke | http://paste.openstack.org/show/584716/ | 16:33 |
| pbourke | its happens randomly in various plays when setup runs | 16:33 |
| pbourke | tried loads of tweaks of ansible ssh settings | 16:33 |
| inc0 | pbourke, never seen anything like this... | 16:34 |
| *** huikang has quit IRC | 16:34 | |
| *** Serlex has quit IRC | 16:34 | |
| pbourke | if I run with -vvv it has something about "debug3: mux_client_read_packet: read header failed: Broken pipe" | 16:35 |
| pbourke | like ssh is just hanging up early or something | 16:36 |
| inc0 | pbourke, yeah, seems to be ssh issue | 16:36 |
| inc0 | or networking | 16:36 |
| *** daneyon has joined #openstack-kolla | 16:37 | |
| *** rstarmer has quit IRC | 16:38 | |
| inc0 | try to tweak up ssh http://askubuntu.com/questions/127369/how-to-prevent-write-failed-broken-pipe-on-ssh-connection | 16:38 |
| inc0 | I've seen this happening during LM | 16:38 |
| openstackgerrit | Waldemar Znoinski (wznoinsk) proposed openstack/kolla: WIP: setting basic structures for iscsi container https://review.openstack.org/381942 | 16:38 |
| *** msimonin has quit IRC | 16:41 | |
| wznoinsk | Jeffrey4l_: https://bugs.launchpad.net/kolla/+bug/1631072 | 16:41 |
| openstack | Launchpad bug 1631072 in kolla "iscsid container: mkdir /sys/kernel/config: operation not permitted" [Undecided,New] | 16:41 |
| zhubingbing | pbourke | 16:42 |
| *** daneyon has quit IRC | 16:42 | |
| Jeffrey4l_ | wznoinsk, never saw that. will check it tomorrow. need go to bed ;0 | 16:48 |
| Jeffrey4l_ | ;) | 16:48 |
| *** msimonin has joined #openstack-kolla | 16:48 | |
| *** zhubingbing has quit IRC | 16:48 | |
| wznoinsk | Jeffrey4l_: cheers, talk to you | 16:48 |
| *** schwicht has quit IRC | 16:48 | |
| *** Pavo has quit IRC | 16:51 | |
| *** neilus__ has quit IRC | 16:51 | |
| *** schwicht has joined #openstack-kolla | 16:51 | |
| *** neilus has joined #openstack-kolla | 16:51 | |
| sbezverk_ | kfox1111: ping me when you have a chance | 16:55 |
| *** Pavo has joined #openstack-kolla | 16:55 | |
| kfox1111 | whats up? | 16:57 |
| sbezverk_ | kfox1111: have you read a couple of messages I sent to you above? | 16:57 |
| kfox1111 | oh. didn't see them. sec | 16:58 |
| kfox1111 | oh you are trying to reduce the time a terminating pod takes? | 16:58 |
| kfox1111 | I think thats because kolla pods don't accept terms nicely and shutdown. | 16:59 |
| sbezverk_ | kfox1111: right and I want to propose to move back to deployment for mariadb and rabbitmq | 16:59 |
| kfox1111 | kubernetes does a 30 second timeout by default unless yo uoverride it in the pod, or fix it to accept a term or preStop hook. | 16:59 |
| sbezverk_ | I noticed that for RC and deployment type of pods they stuck in terminating too, but in parallel new fresh instance is running | 16:59 |
| kfox1111 | -1 | 16:59 |
| inc0 | guys how to debug "Failed to fit in any node" errors? | 17:00 |
| kfox1111 | petsets were designed for stateful services like mariadb or rabbits that have persistance needs. | 17:00 |
| sbezverk_ | inc0 check labeling | 17:00 |
| inc0 | ah, yeah, makes sense | 17:00 |
| kfox1111 | once we get to multinode mariadb/rabbit, we will really benifit from petset's pvc templating. | 17:01 |
| sbezverk_ | kfox1111: I undersatnd but that damn 3-4 minutes terminating phase is a killer | 17:01 |
| *** sdake has joined #openstack-kolla | 17:01 | |
| kfox1111 | sbezverk_: its because its being terminated imporperly. | 17:01 |
| *** vhosakot_ has joined #openstack-kolla | 17:02 | |
| kfox1111 | sbezverk_: we need to add a signal handler or a preStop hook that shuts it down properly. | 17:02 |
| kfox1111 | if you don't want to wait, just add a --now to your delete. | 17:02 |
| kfox1111 | totally unsafe, but emediate. | 17:02 |
| sbezverk_ | kfox1111: what difference does it make if node gets powered off??? | 17:02 |
| sbezverk_ | you can add as many hooks you want but that piece of code already not running | 17:03 |
| *** david-lyle has quit IRC | 17:03 | |
| kfox1111 | not following... | 17:03 |
| *** vhosakot has quit IRC | 17:03 | |
| kfox1111 | maybe we are talking about different thigns. | 17:03 |
| kfox1111 | I'm talking about pod termination. | 17:03 |
| kfox1111 | are you talking about nodes disapearing? | 17:03 |
| sbezverk_ | kfox1111: here is scenario we want to show, compute node with mariadb, it gets powered off | 17:03 |
| sbezverk_ | and mariadb moves to another compute node | 17:03 |
| *** neilus has quit IRC | 17:04 | |
| sbezverk_ | it happens but very long | 17:04 |
| kfox1111 | ah. your tryint to deal with the ceph lock thing. | 17:04 |
| *** MarMat has quit IRC | 17:04 | |
| sbezverk_ | no, locking I clear right after I power down the compute node | 17:04 |
| sbezverk_ | I mean clear manually | 17:04 |
| kfox1111 | ok. and thats not enough to get it to reschedule? | 17:05 |
| sbezverk_ | kfox1111: let me do step by step | 17:05 |
| kfox1111 | or its taking a while to detect the node disapeared and fail over? | 17:05 |
| sbezverk_ | 1. power off compute node with mariadb | 17:05 |
| kfox1111 | oik. please. | 17:05 |
| sbezverk_ | 2. clear ceph locks manually from ceph cluster | 17:05 |
| sbezverk_ | 3. wait for node not ready state takes 20-30 seconds | 17:06 |
| sbezverk_ | 4. wait for pod starts terminating with new timers takes less time | 17:06 |
| *** srwilkers has joined #openstack-kolla | 17:06 | |
| sdake | morning again | 17:06 |
| *** unicell has quit IRC | 17:07 | |
| sdake | boy i dislike dentistry | 17:07 |
| sbezverk_ | 5. wait for pod gets actualli terminated takes lots of time | 17:07 |
| sdake | i mean i like the concept | 17:07 |
| sdake | just not when applied to me | 17:07 |
| sbezverk_ | 6. wait for pod to get started on new node very fast as soon as terminating phase is gone | 17:07 |
| kfox1111 | sbezverk_: ok. and because petsets only let one exist, it has to wait until its done terminating... | 17:08 |
| sbezverk_ | the worst part for us is 4 | 17:08 |
| sbezverk_ | exactly | 17:08 |
| sbezverk_ | rc and deployment starts new pod and that is, but not fot petsets | 17:08 |
| kfox1111 | the ultimate solution to this is galera. | 17:08 |
| kfox1111 | then taking a while for the one pod to come back up doesn't matter much. | 17:08 |
| kfox1111 | and there is no downtime what so ever. | 17:08 |
| sdake | inc0 about? | 17:10 |
| sbezverk_ | kfox1111: maybe but not sure if it is doable before the summit, I would really like to explore deployment for mariadb and see if it can at least temporarely improve things | 17:10 |
| inc0 | sdake, yeah | 17:10 |
| inc0 | we kicked off thread analysis | 17:10 |
| sdake | inc0 kfox1111 and i had a convo last night about repo split | 17:10 |
| kfox1111 | sbezverk_: yeah, thats all fine and all. but I'm an operator that doesnt appreciate too much hand wavy hackery for demo's. I now there are other ops in the same boat. | 17:11 |
| sdake | inc0 he said you are a f an of keepingkubernetes workflow in kolla-ansible repo | 17:11 |
| kfox1111 | we shouldn't go too far out of our way to do a poor archetecture just to do a demo that looks sort of good but that ops can see through. | 17:11 |
| sdake | inc0 i dont like that idea | 17:11 |
| kfox1111 | all the demos for kubernetes so far only ever just did stuff like delete pod, whatch it come back. | 17:11 |
| kfox1111 | but never really did it seamless. | 17:12 |
| kfox1111 | and people eat that up. | 17:12 |
| inc0 | sdake, why? | 17:12 |
| kfox1111 | so we should be able to do a really good demo, with what we have, and still make a good impresion. | 17:12 |
| sdake | dont crater the arch to make a rockin demo | 17:12 |
| sbezverk_ | kfox1111: well if 10 minutes is acceptabel time of ops for a service restoration on a ndoe failure | 17:12 |
| inc0 | we have dep on kolla-ansible anyway | 17:12 |
| sbezverk_ | then we could show it as is | 17:12 |
| inc0 | I don't want to have ansible playbooks all over the place | 17:13 |
| sdake | inc0 welcome to repo split madness | 17:13 |
| inc0 | and kfox doesn't want to have ansible in kolla-k8s...which makes sense | 17:13 |
| sdake | i dont want it in kolla-ansible | 17:13 |
| inc0 | why? | 17:13 |
| sdake | kolla-ansible is for orchestrating bare metal with ansible | 17:13 |
| sdake | not kubernetes | 17:13 |
| sdake | we have a proposal as follows | 17:13 |
| kfox1111 | sbezverk_: we can have the script do 3 things. power off the node, kill the ceph lock, and force delete the petset. and we can tell people that its a k8s issue if asked, and they are actively working on it for 1.5 and point them at the github issue. | 17:14 |
| inc0 | it also deploys docker with ansible | 17:14 |
| sdake | 1. rename kolla-kubernetess to kolla-kube | 17:14 |
| inc0 | or bifrost with ansible | 17:14 |
| kfox1111 | sbezverk_: most ops have to recover by hand today. any auto recovery is a huge benifit. | 17:14 |
| sdake | yes, it stands alone as a peice of work | 17:14 |
| sdake | make a kolla-kube-ansible repo | 17:14 |
| sdake | make a kolla-ansible repo | 17:14 |
| sdake | keep kola named as is | 17:14 |
| sdake | kolla repo = containers + buidl scripts | 17:14 |
| inc0 | ehh...you want to grow repos exponentially? | 17:14 |
| sdake | e.g. the core | 17:14 |
| inc0 | first, this will be temp | 17:14 |
| sdake | kfox1111 how about a little help on your idea here... | 17:15 |
| inc0 | as we aim to have k8s native thing when it becomes possible | 17:15 |
| Daviey | Marmat wanted me? | 17:15 |
| kfox1111 | sdake: sure. | 17:15 |
| sdake | inc0 i'd recommend reading logs toget up to speed on the thinking | 17:15 |
| inc0 | guys, it doesn't make a remote sense imho. We have logic in kolla-ansible we could reuse | 17:16 |
| rhallisey | O.o | 17:16 |
| kfox1111 | inc0: the issue is, kolla-kubernetes is a set of templates for standing up stuff in k8s, just like kolla is a set of containers for packaging up openstack. | 17:16 |
| rhallisey | we need to have the argument in person XD | 17:16 |
| *** neilus has joined #openstack-kolla | 17:16 | |
| inc0 | more than happy to have it. | 17:16 |
| kfox1111 | workflow is a seperate thing. and its quite religiously charged. | 17:16 |
| inc0 | can I bring knife? | 17:16 |
| sdake | i need to know now what is planned | 17:16 |
| kfox1111 | :) | 17:16 |
| kfox1111 | would a hangout be better for the converastion? | 17:17 |
| inc0 | kfox1111, workflow ideally will be handled by k8s natively...someday | 17:17 |
| sdake | becaue i am presenting stuff at summit on it :) | 17:17 |
| inc0 | before that we need something | 17:17 |
| sdake | right in in the meantime we have this repo off to the side | 17:17 |
| sdake | which is the workflowbits | 17:17 |
| rhallisey | sdake, there will be ansible bits the will deploy | 17:17 |
| inc0 | having separate ansible playbook in kolla-ansible makes sense - we already use kolla-ansible genconfig | 17:17 |
| inc0 | we *need* parts of it | 17:17 |
| kfox1111 | inc0: I think there will always be at least a little bit of workflow that will be in external systems, lilke puppet/chef, etc. enterprises will demand that. :/ | 17:17 |
| rhallisey | sdake, it will sit in the same repo until post summit | 17:17 |
| rhallisey | where we will consider making any changes | 17:18 |
| sdake | rhallisey i understand that | 17:18 |
| kfox1111 | but I agree, if k8s can handle worflw bits, it should. | 17:18 |
| inc0 | if we separate workflow to yet another repo | 17:18 |
| kfox1111 | but openstack's workflow is hugely complicated. | 17:18 |
| sdake | rhallisey however we have already made a deicsionvia vote to split the repos | 17:18 |
| rhallisey | sdake, yup | 17:18 |
| kfox1111 | it may be years before k8s can handle it. | 17:18 |
| rhallisey | sdake, that's still needed | 17:18 |
| inc0 | we'll need kolla-ansible to genconfig, kolla-kube-ansible to workflow and kolla-kube to well..kube | 17:18 |
| inc0 | 3 projects | 17:18 |
| sdake | rhallisey which is still needed | 17:18 |
| sdake | rhallisey the vote has happened | 17:18 |
| rhallisey | sdake, the repo split | 17:19 |
| sdake | rhallisey its a done deal | 17:19 |
| inc0 | why you don't want another file in kolla-ansible? | 17:19 |
| sdake | and i am presenting it at summit | 17:19 |
| rhallisey | I meant the repo split not the vote | 17:19 |
| sdake | inc0 its more like 100's of files | 17:19 |
| inc0 | no sdake | 17:19 |
| kfox1111 | inc0: config and workflow are seperate. | 17:19 |
| inc0 | kfox1111, only because we made it so | 17:19 |
| kfox1111 | I may hand config my stuff, but then want to use the ansible worflow, | 17:19 |
| inc0 | if we could access roles in kolla-ansible | 17:19 |
| rhallisey | sdake, we can't put kolla-k8s anisble bits into kolla-ansible without a solid plan in place | 17:19 |
| inc0 | we can just import config.yml and it's done | 17:20 |
| kfox1111 | or I might use genconfig to build the config, but manually deploy or use chef. | 17:20 |
| sdake | kolla-ansible should stand on its own | 17:20 |
| rhallisey | there is no such thing at this point | 17:20 |
| sdake | just as kolla stands on its own | 17:20 |
| sdake | or a repo split makes no sense | 17:20 |
| inc0 | sdake, it will stand on it's own, kolla-k8s will not stand on it's own | 17:20 |
| sdake | kollakube should stand on its own for same reason | 17:20 |
| rhallisey | that idea is only in the ealry stages | 17:20 |
| rhallisey | sdake, I agree with that logic | 17:20 |
| inc0 | if we would decouple configs to kolla-core | 17:20 |
| inc0 | then we don't need kolla-ansible for kolla-k8s | 17:21 |
| inc0 | and then it makes sens | 17:21 |
| inc0 | ee | 17:21 |
| sdake | then the ansible workflow related to kuberentes is going in the kube repo | 17:21 |
| sdake | not the ansible repo | 17:21 |
| kfox1111 | inc0: yeah, that woudl be idea. | 17:21 |
| kfox1111 | ideal. | 17:21 |
| inc0 | kfox1111, that was the plan, just didn't happen now | 17:21 |
| inc0 | what's wrong about current solution tho? dir workflow/ansible | 17:21 |
| *** neilus has quit IRC | 17:21 | |
| kfox1111 | yeah. I get that. I'm going to help with that too if I can. | 17:21 |
| kfox1111 | inc0: perception. | 17:22 |
| inc0 | is word ansible so dirty that even having these files on disk prevents people from using kolla? | 17:22 |
| sdake | inc0 kfox1111 doesnt' want workflow in k8s | 17:22 |
| kfox1111 | the huge stick around breaking ansible out to its own repo. | 17:22 |
| sdake | and it definately doesn't belong in kolla-ansible | 17:22 |
| inc0 | we need workflow in k8s | 17:22 |
| kfox1111 | it would be better not to have ansible code land in kolla-kubernetes and pull it out later. | 17:22 |
| kfox1111 | from a perception standpoint. | 17:22 |
| inc0 | kfox1111, you don't want to end up in cross-repo dependency hell | 17:22 |
| kfox1111 | I agree, rwe really need the workflow. | 17:22 |
| inc0 | look how well it worked between kolla and kolla-k8s | 17:22 |
| sdake | ok well lets focuso on techncial roblesm not perception problems | 17:22 |
| kfox1111 | whats there to have hell about? the ansible stuff just calls the cli. | 17:23 |
| inc0 | agree sdake | 17:23 |
| kfox1111 | the cli's api is very simple. :) | 17:23 |
| inc0 | if somebody think that these yamls on disk are so dirty, I don't really care about this person | 17:23 |
| inc0 | they can rm -rf the dir after cloning it | 17:23 |
| kfox1111 | inc0: thats a lot of devs. :/ | 17:23 |
| sdake | inc0 carefu lwti that logic :) | 17:23 |
| inc0 | issue is, it will be completely optional | 17:23 |
| sdake | inc0 yamls (ansible) on disk seems to be a sore point of contention even though epoplecould just delete it | 17:23 |
| sdake | inc0 no shit i argued that in the original review | 17:24 |
| inc0 | it's like saying that I'm not going to use cinder because they have netapp driver in repo and I don't like netapp | 17:24 |
| sdake | and then was told "I just destroyed my strawman argument with my argument" | 17:24 |
| kfox1111 | inc0: yeah. thats been kolla's party line for a long time. and its bit kolla hard I think. :/ | 17:24 |
| sdake | it hasn't bit kollat a ll | 17:24 |
| inc0 | it didn't | 17:24 |
| sdake | people either like ansible or htey dont | 17:24 |
| sdake | if they dont - there is no option for them that is viable imo | 17:24 |
| sdake | besdies RYO | 17:24 |
| kfox1111 | people are really religious about config management tools. so much so that they stop thinking reasonably about things like, if yo udon't want to use the optional stuff, dont use it. | 17:25 |
| kfox1111 | they spawn whole noew projects instead. :/ | 17:25 |
| inc0 | kfox1111, issue is, it will always be totally optional | 17:25 |
| *** david-lyle has joined #openstack-kolla | 17:25 | |
| inc0 | if they want to be religious, let them in their zeal write puppet code for kolla-k8s | 17:25 |
| kfox1111 | "your confusing the issue with logic" :) | 17:25 |
| inc0 | then they can put it into dir worflow/puppet | 17:25 |
| kfox1111 | its not a logicical issue. :) | 17:26 |
| sdake | we focus on solving technical problems kfox1111 | 17:26 |
| sdake | the technical problem (the reason we are splitting the repos) is because they are not consumable by kolla-kubernetes in an easy fashion | 17:26 |
| kfox1111 | sdake: there is no reason to split repos on a purely technical reason. | 17:26 |
| inc0 | kfox1111, lets not try to make world that much better for people who aren't here | 17:26 |
| *** MarMat has joined #openstack-kolla | 17:26 | |
| inc0 | if somebody *really* need that and having ansible on disk is so dirty that they can't use kolla | 17:27 |
| kfox1111 | inc0: no, I want to make it better for all of us, by not turning off devs. | 17:27 |
| inc0 | let them come here and shout out, we can discuss then | 17:27 |
| sdake | we can't be all things to all people | 17:27 |
| kfox1111 | and ops. | 17:27 |
| sdake | unless all people come up to implement all things | 17:27 |
| kfox1111 | that makes it better for us, as we will have mor eusers, and more devs. | 17:27 |
| inc0 | kfox1111, and 100 pararell workstreams | 17:27 |
| inc0 | which might in effect slow our progress | 17:27 |
| sdake | inc0 starting to get the picture on repo splits :) | 17:27 |
| kfox1111 | inc0: thats not how it works. they just spawn a fuelccp or a openstack-ansible. | 17:27 |
| kfox1111 | then we suffer by fewer devs. | 17:28 |
| inc0 | sdake, I had this picture from day one, that's why I was opposed to it in Austin | 17:28 |
| sdake | kfox1111 fwiw openstack-ansible was ahead of us | 17:28 |
| inc0 | kfox1111, they use openstack-ansible because they hate ansible?:)P | 17:28 |
| sdake | inc0 i didn't get that impression | 17:28 |
| inc0 | not that way ahead of us sdake | 17:28 |
| inc0 | I bet we'll outrun them in release or two in terms of deployments | 17:28 |
| sdake | inc0 on a totall ydifferent development path way ahead inc0 | 17:28 |
| sdake | now we pretty much have similar features or kolla has better | 17:29 |
| sdake | anyway that is irrelevant | 17:29 |
| sdake | just dont say that out loud, you will ook like a dummy :) | 17:29 |
| sdake | fuelccp - fair to say :) | 17:29 |
| inc0 | kfox1111, fuel-ccp will do a lot to go native k8s | 17:29 |
| inc0 | and that's cool, we should move towards native k8s too | 17:29 |
| sdake | ok dudes, we all knwo native k8s is the way to go | 17:29 |
| inc0 | when it's possible | 17:29 |
| inc0 | before that, let's make a ansible prothesis | 17:30 |
| sdake | in the meantime | 17:30 |
| sdake | what inc0 said | 17:30 |
| inc0 | put it into dir | 17:30 |
| inc0 | in kolla-k8s or kolla-ansible | 17:30 |
| sdake | but its not going into kolla-ansible ;) | 17:30 |
| inc0 | and rm -rf when we no longer need it | 17:30 |
| sbezverk_ | inc0: it would be great to have this prothesis well detachable and replacable ;-) | 17:30 |
| inc0 | I'm not going to fight that battle, for me it makes sense either way | 17:30 |
| kfox1111 | kolla-ansible is preferable to kolla-kubernetes. | 17:30 |
| *** g3ek has quit IRC | 17:30 | |
| sdake | kfox1111 - it is better to keep one working thing together | 17:31 |
| sdake | i understand where you want to take kollakube | 17:31 |
| kfox1111 | I'd prefer kolla-kubernetes to be ansible free. its pretty much there already. just a few remaining bits to go. | 17:31 |
| inc0 | putting it into kolla-ansible will remove need of genconfig and fix this perception issue | 17:31 |
| sdake | it took us 3 years to get to that point in kolla itself | 17:31 |
| sdake | and we still have't done the repo split | 17:31 |
| inc0 | but will hinder the ability to develop kolla-k8s as kolla-ansible cores will have to ack any change | 17:31 |
| *** unicell has joined #openstack-kolla | 17:31 | |
| sdake | lets not try to solve that problem right up front | 17:31 |
| kfox1111 | I agree with inc0. | 17:31 |
| rhallisey | I don't want to solve it upfront either | 17:31 |
| rhallisey | the project is too young | 17:32 |
| kfox1111 | the k8s ansible workflow is really simple. I don't think the kolla core's will slow it down much. | 17:32 |
| *** haplo37 has quit IRC | 17:32 | |
| inc0 | kolla-k8s path will leave some bad impressions (which I'm ok with tbh) and still genconfig from kolla-ansible until we move config templates to kolla-core | 17:32 |
| sdake | kfox1111 it doesn't belong in that repo | 17:32 |
| sbezverk_ | kfox1111 +1 we do not need to deal with crazy inventory | 17:32 |
| kfox1111 | sdake: it doens't be long in kolla-kubernetes either. :/ | 17:32 |
| sdake | kfox1111 moreso then kolla-ansible. | 17:32 |
| sdake | kfox1111 your lookign at the end state | 17:33 |
| sdake | lets look at the stpes to get to the end state | 17:33 |
| kfox1111 | sdake: both end state, and intermediate perception. :/ | 17:33 |
| inc0 | ok, let's just keep status quo and discuss this in Barcelona ok? | 17:33 |
| sdake | perception =NOT OUR PROBLEM | 17:33 |
| inc0 | if somebody will be rebuked by it...tough luck. we need workflow asap | 17:33 |
| kfox1111 | inc0: do we wait on workflow till barcelona then? | 17:33 |
| sdake | kfox1111 workflow in kolla-k8s | 17:33 |
| rhallisey | kfox1111, we have session in barcelona to discuss the path forward | 17:33 |
| kfox1111 | sdake: it is our problem, we can choose to ignore it though. :/ | 17:33 |
| sdake | we can sort out the history nd repo naming later | 17:33 |
| inc0 | no, we do what Ryan, Qin and Justin are doing now | 17:33 |
| *** g3ek has joined #openstack-kolla | 17:34 | |
| rhallisey | this is where we can figure it out | 17:34 |
| inc0 | kfox1111, what we need to do is to encourage people to not make religious battles where they are none to be had | 17:34 |
| *** haplo37 has joined #openstack-kolla | 17:34 | |
| kfox1111 | inc0: my preference is to never land it it kolla-kubernetes. by dealying, it make s that decision defacto. | 17:34 |
| sdake | inc0 good luc k with that due | 17:34 |
| sdake | 2000 years of history says you will fail :) | 17:34 |
| inc0 | if they like puppet, let them create dir workflow/puppet and be done with it | 17:34 |
| kfox1111 | inc0: thats impossible. | 17:34 |
| inc0 | why kfox1111 ? | 17:34 |
| kfox1111 | why are we splitting kolla-ansible from kolla? | 17:35 |
| kfox1111 | why all the fighting over it if what you said is true? | 17:35 |
| inc0 | because we do have an alternative | 17:35 |
| rhallisey | beacuse there are docker containers and deployment | 17:35 |
| inc0 | and we don't want to have 1st and 2nd class citizens | 17:35 |
| rhallisey | it's in the manifesto | 17:35 |
| inc0 | we were thinking of putting dir into kolla called "kubernetes" remember? | 17:36 |
| kfox1111 | the same is true of workflow. | 17:36 |
| kfox1111 | we don't want to have second class worflows. | 17:36 |
| inc0 | it was good idea, we've chosen to not do this because ack from kolla core would hinder kolla-k8s dev | 17:36 |
| kfox1111 | if there is one in tree, it will be considered the blessed one, and turn off others from contributing new ones. | 17:36 |
| inc0 | that was the reason | 17:36 |
| sdake | kfox1111 the workflows for k8s will remain second class until they have feature parity with the kolla-ansible workflows | 17:36 |
| inc0 | not the case for workflow | 17:36 |
| inc0 | we're not splitting workflows until there is another one | 17:37 |
| sdake | inc0 kubernetes is "another one" | 17:37 |
| inc0 | until we have community called kolla-k8s-ansible | 17:37 |
| *** tonanhngo has quit IRC | 17:37 | |
| inc0 | sdake, workflows in k8s | 17:37 |
| sdake | wfm | 17:37 |
| sdake | workflows in k8s = go in k8s repo | 17:37 |
| inc0 | then this community can stand for themselves and we will discuss | 17:37 |
| sdake | whatever that workflow tech may be | 17:37 |
| HyperJohnGraham_ | Hi all | 17:38 |
| sdake | sup HyperJohnGraham_ | 17:38 |
| inc0 | right, k8s repo not new k8s-ansible repo | 17:38 |
| kfox1111 | what I want to do is, ensure there is no excuse for tripleo, fuel, or other tool not to use kolla-kubernetes for the deployment piece with reguars to wrokflow and configmanagement | 17:38 |
| sdake | ok that wfm - i was working on answering kevin's issue as well | 17:38 |
| HyperJohnGraham_ | sitting in meetings till noon | 17:38 |
| otavio | I tried to sign the CLA but the system seems out. Is there someone who could help on this? | 17:38 |
| *** athomas has quit IRC | 17:38 | |
| inc0 | kfox1111, if you know anyone who would like to pick up creation of new community of puppet-workflow-lovers, let them come here;) | 17:38 |
| kfox1111 | they may have other reasons, but the reasons should be really clear why at that point. | 17:38 |
| inc0 | I assume you're not biased agianst ansible yourself? | 17:39 |
| sdake | otavio try #openstack-infra | 17:39 |
| kfox1111 | inc0: I may be able to get some chef devs at some point. my group is riddled with them. | 17:39 |
| rhallisey | kfox1111, I think we need to dev in kolla-k8s | 17:39 |
| kfox1111 | and they are very religious about it. | 17:39 |
| rhallisey | then conisder kolla-kube-anisble | 17:39 |
| *** dwalsh has joined #openstack-kolla | 17:39 | |
| inc0 | kfox1111, make them start with kolla-chef | 17:39 |
| sdake | lets not take on any more work at this point | 17:39 |
| inc0 | if they want to use k8s bits there - fine by me | 17:39 |
| sdake | we got enough stuff to sort out | 17:39 |
| kfox1111 | they suffere through bad deployment tools because tis what they know, and the blessed way of doing things, | 17:39 |
| rhallisey | there hasn't been a release yet for it so we don't have backports or anything | 17:39 |
| kfox1111 | rather then use better tools. | 17:39 |
| inc0 | but let's not solve problems which might never appear | 17:39 |
| rhallisey | we can stil be felexible here | 17:39 |
| sdake | inc0 ya thats called tiliting at windmills :) | 17:40 |
| sdake | phantom threats | 17:40 |
| sdake | no reason to worry about those | 17:40 |
| kfox1111 | what is so hard about putting some ansible bits in a seperate repo? | 17:40 |
| inc0 | kfox1111, cross-repo deps | 17:40 |
| inc0 | release process | 17:40 |
| rhallisey | kfox1111, it changes the project | 17:40 |
| inc0 | core team management | 17:40 |
| rhallisey | which we don't want to do until we have to | 17:40 |
| sdake | right | 17:40 |
| inc0 | deployment workflow (need to install 2 instead of 1) | 17:40 |
| kfox1111 | thats pretty easy to deal with at this point I think. | 17:41 |
| kfox1111 | cores = cores on both repos. | 17:41 |
| inc0 | no, not easy | 17:41 |
| inc0 | for how long? | 17:41 |
| kfox1111 | cross repo deps, its kollakube... | 17:41 |
| sdake | agree with inc0 its not easy | 17:41 |
| sdake | its a permanent choice | 17:41 |
| inc0 | also, cross-repo deps are pita | 17:41 |
| kfox1111 | if the deps break, tis bad. we're breaking an api. | 17:41 |
| kfox1111 | we need to ensure api anyway. | 17:41 |
| kfox1111 | that encurages us not to break api. | 17:41 |
| inc0 | I'd rather have this one be done by CI | 17:41 |
| kfox1111 | release process, yeah. its an extra few steps. I'll by that. | 17:41 |
| inc0 | not pita in gerrit | 17:42 |
| *** ankush has quit IRC | 17:42 | |
| sdake | kfox1111 inc0 is correct, it is not as easy as just making a repo | 17:42 |
| inc0 | 2 different CI confs | 17:42 |
| sdake | lots of repos are an anti-pattern imo | 17:42 |
| kfox1111 | I understand the pain of ci. :) | 17:42 |
| sdake | repo splits happen when the current repo structure is untenable | 17:42 |
| kfox1111 | been dealing with it a lot. :) | 17:42 |
| sdake | we haven't really reached that point | 17:42 |
| sdake | but we are coming to it | 17:42 |
| inc0 | kfox1111, it's sum of all pains | 17:42 |
| kfox1111 | if we split it, I'll setup the gate for it. how about that? :) | 17:43 |
| inc0 | why we don't want separate repo unless there is *really* good reason for it | 17:43 |
| *** tonanhngo has joined #openstack-kolla | 17:43 | |
| sdake | setup gate is cheap | 17:43 |
| sdake | gate maintenance is not cheap | 17:43 |
| inc0 | kfox1111, no, let's get kolla-k8s working on *any* workflow before | 17:43 |
| sdake | kfox1111 process the above two statements with your understanding of what it takes to setup thegate ;) | 17:43 |
| inc0 | let's get community of people advocating for split | 17:44 |
| kfox1111 | inc0: that will probably be done today. | 17:44 |
| *** tonanhngo has quit IRC | 17:44 | |
| inc0 | there is good chance that this community won't appear before k8s grow native support | 17:44 |
| inc0 | and in fact we can have semi-native support today | 17:44 |
| inc0 | if we write kolla-start-k8s to understand deps, much like mirantis does it in fuel-ccp | 17:44 |
| inc0 | or SAP | 17:44 |
| sdake | ok so some definitions are in order | 17:45 |
| rhallisey | kfox1111, the workflow bits are all in a single repo at the moment | 17:45 |
| kfox1111 | inc0: possible. also possible they just wont adopt kolla until that happens. it may be come self fulfiling. | 17:45 |
| sdake | kolla = containers + build (our core stuff) | 17:45 |
| rhallisey | that separate repo essentially exists in that dir | 17:45 |
| inc0 | kfox1111, I'm not writting chef code for kolla-k8s, will you? | 17:45 |
| rhallisey | s/repo/dir/ | 17:45 |
| sdake | kolla-kube = everything related to kolla + kubernetes | 17:45 |
| kfox1111 | nope. but I don't want to turn off the devs that would write a chef version. | 17:45 |
| rhallisey | we will keep the workflow code seperate in the workflow dir until we deem it worth to split | 17:45 |
| sdake | kolla-ansible = everything related to orchestration for kolla-ansible API | 17:45 |
| inc0 | if they will feel bad, let them come here and talk to us | 17:46 |
| sdake | kfox1111 phantoms imo :) | 17:46 |
| kfox1111 | kolla-kube= the database of kolla-kube templates, ad a simple tool for managing them. | 17:46 |
| sdake | kfox1111 that is the end state | 17:46 |
| kfox1111 | config/workflow is a seperate thing. | 17:46 |
| sdake | it can be a seprate thing in the same repo | 17:46 |
| inc0 | ansible is separate thing in same repo as containers | 17:46 |
| kfox1111 | inc0: it is unknowable how many wil just turn and run rather then come and complain. but, | 17:46 |
| kfox1111 | jusdging from my operator experience, | 17:47 |
| kfox1111 | 1 person complains about an issue, and 9 just ignore it. | 17:47 |
| kfox1111 | if you hear something from someone, you have to assume there are many more behind staying quite. | 17:47 |
| inc0 | kfox1111, juding from my operator experience, ops are clever people, most of them won't care religiously about having a dir on disk | 17:47 |
| sdake | kfox1111 yup thatis a reality | 17:47 |
| inc0 | again, case of cinder and netapp driver | 17:47 |
| *** schwicht has quit IRC | 17:47 | |
| inc0 | nobody cares about having it there besides people who uses it | 17:48 |
| *** ankush has joined #openstack-kolla | 17:48 | |
| inc0 | call it a workflow driver | 17:48 |
| inc0 | change dir name to workflow_drivers or workflow_plugins | 17:48 |
| inc0 | that should deal with perception | 17:48 |
| sdake | inc0 should but doesn't | 17:48 |
| sdake | we aren't here to solve perception problems, technical problems.... | 17:48 |
| kfox1111 | hmm... | 17:49 |
| inc0 | what I'm saying, that's what we can do without hindering our ability to deliver | 17:49 |
| sdake | kolla-kubernetes is so green nobody will really care about perception issues | 17:49 |
| inc0 | ability to deliver >>> perception | 17:49 |
| rhallisey | agreed | 17:49 |
| kfox1111 | seperate repo > /workflow_drivers > /workflow > /ansible | 17:49 |
| sdake | ya need to focus on execution rather then politicizing kolla even more then has already beendone | 17:49 |
| kfox1111 | perception is a big deal to new prjects though. if your understaffed. | 17:50 |
| inc0 | kfox1111, we'll deal with staff | 17:50 |
| kfox1111 | we need to make it as easy as possible to get additional developers. | 17:50 |
| sdake | kfox1111 working on getting that fixed | 17:50 |
| inc0 | don't worry | 17:50 |
| *** huikang has joined #openstack-kolla | 17:50 | |
| inc0 | just give us time, it's very first summit for this project | 17:50 |
| inc0 | when it actually exist | 17:50 |
| inc0 | s | 17:50 |
| sdake | with 3 months of dev time... | 17:50 |
| kfox1111 | with at least 3 other projects hot on our heels. :/ | 17:51 |
| kfox1111 | its a critical time for the project. | 17:51 |
| kfox1111 | one thing I've leared about tech... | 17:51 |
| inc0 | yes, so let's focus on delivering it asap | 17:51 |
| inc0 | this is how you win | 17:51 |
| kfox1111 | its not always the best that wins. | 17:51 |
| kfox1111 | its the first to get the general good perception. | 17:51 |
| inc0 | not by making smoke and mirrors and making everyone happy without result at the end | 17:51 |
| kfox1111 | and it snowballs into a clear winner. | 17:52 |
| sdake | right, and kolla has a fantastic perception ingeneral | 17:52 |
| *** neilus has joined #openstack-kolla | 17:52 | |
| sdake | all that done - IN ONE REPO | 17:52 |
| rhallisey | kfox1111, new devs will come | 17:52 |
| inc0 | kfox1111, no, it will move religious war into kolla community | 17:52 |
| kfox1111 | rhallisey: yeah, but will it be enough at the right time? | 17:52 |
| inc0 | when chef people will be stopped by changes in shared repo that ansible people will need to review | 17:53 |
| inc0 | look how much of a problem that was between kolla and kolla-k8s | 17:53 |
| rhallisey | kolla has 3 devs for the first ~8 months | 17:53 |
| sdake | chef not going into kolla-ansible | 17:53 |
| rhallisey | look at it now | 17:53 |
| sdake | the idea of deliverables is they "stand alone" | 17:53 |
| sdake | (in the tc sense of deliverables in projects.yaml) | 17:53 |
| harlowja | qq, for folks, cause i forget, where are all the config.json files in kolla repo | 17:54 |
| sdake | harlowja all over the ansible dir tree | 17:54 |
| kfox1111 | rhallisey: I wasn't turned off by ansible. I bothered to learn a lot about it as I do use it a bit already. If I was more in the chef camp, I might have been much more turned off. | 17:54 |
| inc0 | harlowja, https://github.com/openstack/kolla/blob/master/ansible/roles/cinder/templates/cinder-backup.json.j2 | 17:54 |
| inc0 | for exampel | 17:54 |
| rhallisey | harlowja, ansible/roles/<service>/defaults/config.json | 17:54 |
| harlowja | thx, thought i looked there, lol | 17:54 |
| rhallisey | nvm it's in templates | 17:54 |
| *** huikang has quit IRC | 17:54 | |
| *** david-lyle has quit IRC | 17:55 | |
| sdake | find . -name config.json ;) | 17:55 |
| harlowja | ya ya | 17:55 |
| inc0 | kfox1111, we all know ansible most | 17:55 |
| harlowja | :-P | 17:55 |
| rhallisey | kfox1111, I get your perspective. But we'ere working in the *new* areas of deployment | 17:55 |
| inc0 | we'll be turned off by chef, so we won't write code | 17:55 |
| *** huikang has joined #openstack-kolla | 17:55 | |
| rhallisey | people will show up to see what it is | 17:55 |
| inc0 | if someone would like to do it, happy to accept them to community | 17:55 |
| inc0 | let them make a dir in workflows. | 17:55 |
| rhallisey | we can show multiple examples of deloyment | 17:55 |
| rhallisey | which we have curently | 17:55 |
| rhallisey | we have the CLI deloyment and a workflow | 17:56 |
| kfox1111 | inc0: I'm not asking for anyone here to write code for any other config management tool. | 17:56 |
| sdake | the reason we are splitting the repos is to deliver on the "Converged datacenter" | 17:56 |
| inc0 | kfox1111, but if we would, that would send a clear message that ansible is not only workflow | 17:56 |
| sdake | thats the reason kolla-k8s exists as well | 17:56 |
| kfox1111 | just trying to make it as level a playing field technically, and perceptially for ayone wanting to do so. | 17:56 |
| inc0 | we can make a dir for chef today | 17:56 |
| inc0 | :) | 17:56 |
| *** neilus has quit IRC | 17:56 | |
| rhallisey | kfox1111, yes agreed. It's up to the cores though | 17:56 |
| rhallisey | and the architure we adapt | 17:57 |
| rhallisey | we need to remain open and keep multiple entrpoints available | 17:57 |
| inc0 | anyway, kfox1111 can we please agree to have this discussion on summit? | 17:57 |
| kfox1111 | keeping the kolla-kubernetes stuff seperate from the ansible stuff, ensures that the api is at a level playing field. | 17:57 |
| inc0 | you can bring your chef collegues to table so we can have educated discussion with another side too | 17:57 |
| kfox1111 | inc0: I can't attend the summit. | 17:57 |
| inc0 | ah sorry, forgot | 17:57 |
| inc0 | invite them here then>? | 17:57 |
| kfox1111 | and what do we do until the summit? | 17:57 |
| inc0 | kfox1111, carry on with plan | 17:58 |
| *** neilus has joined #openstack-kolla | 17:58 | |
| inc0 | I'm not making new repo before summit...I'll have to make one already | 17:58 |
| rhallisey | kfox1111, stick with the plan. It will be discussed at the summit sessions we have | 17:58 |
| kfox1111 | until the summit, does the code go in kolla-ansible, or kolla-kubernetes? | 17:58 |
| sdake | that plan shouldl not include putting k8s workflow in kolla-ansible | 17:58 |
| rhallisey | kfox1111, kolla-k8s | 17:58 |
| sdake | inc0 i can handle the repos | 17:58 |
| awiddersheim | speaking of netapp | 17:58 |
| sdake | but also not prior to summit | 17:58 |
| inc0 | kfox1111, same as Ryan is doing now. kolla-k8s | 17:58 |
| awiddersheim | anyone have experience iwth doing netapp with kolla and NFS? | 17:58 |
| kfox1111 | k. sound like I'm overruled. thats fine. I've said my peace. :) | 17:59 |
| inc0 | awiddersheim, nope:( | 17:59 |
| awiddersheim | or NFS in general I guess | 17:59 |
| awiddersheim | with kolla | 17:59 |
| sdake | kfox1111 i think separate repo for workflow for kolla-kube works fine | 17:59 |
| rhallisey | kfox1111, thank you for you opinion :) | 17:59 |
| sdake | kfox1111 but lets get the workflow working first ;) | 17:59 |
| rhallisey | kfox1111, I agree we need to disccus it | 17:59 |
| sdake | and make sure kolla-k8s can stand alone | 17:59 |
| *** huikang_ has joined #openstack-kolla | 17:59 | |
| *** huikang has quit IRC | 17:59 | |
| openstackgerrit | Otavio Salvador proposed openstack/kolla: Update Magnum dockerfiles for formatting https://review.openstack.org/383167 | 17:59 |
| openstackgerrit | Otavio Salvador proposed openstack/kolla: Add support for magnum ubuntu binary containers https://review.openstack.org/383168 | 17:59 |
| openstackgerrit | Otavio Salvador proposed openstack/kolla: Fix tar command path for magnum conductor using ubuntu https://review.openstack.org/383169 | 18:00 |
| openstackgerrit | Otavio Salvador proposed openstack/kolla: Fix typos in Kolla https://review.openstack.org/383170 | 18:00 |
| sdake | inc0 which new repo are you amking | 18:00 |
| openstackgerrit | Otavio Salvador proposed openstack/kolla: Use packaged pip instead of easy_install https://review.openstack.org/383171 | 18:00 |
| openstackgerrit | Otavio Salvador proposed openstack/kolla: Improve Vagrants bootstrap.sh proxy support https://review.openstack.org/383172 | 18:00 |
| inc0 | I'd rather explore way to use k8s native | 18:00 |
| openstackgerrit | Otavio Salvador proposed openstack/kolla: Refer to operator VM as primary VM https://review.openstack.org/383173 | 18:00 |
| openstackgerrit | Otavio Salvador proposed openstack/kolla: Set internal vip to vagrant private network https://review.openstack.org/383174 | 18:00 |
| openstackgerrit | Otavio Salvador proposed openstack/kolla: Remove unused SUPPORT_NODE variable https://review.openstack.org/383175 | 18:00 |
| openstackgerrit | Otavio Salvador proposed openstack/kolla: Only keep localhost for 127.0.0.1 https://review.openstack.org/383176 | 18:00 |
| openstackgerrit | Otavio Salvador proposed openstack/kolla: Connect to remote hosts as root https://review.openstack.org/383177 | 18:00 |
| openstackgerrit | Otavio Salvador proposed openstack/kolla: Use MAC address for looking for DHCP leases for Vagrant https://review.openstack.org/383178 | 18:00 |
| openstackgerrit | Otavio Salvador proposed openstack/kolla: Fix docker daemon proxy support in vagrant https://review.openstack.org/383179 | 18:00 |
| openstackgerrit | Otavio Salvador proposed openstack/kolla: Vagrant plugin check https://review.openstack.org/383180 | 18:00 |
| kfox1111 | sdake: worklow working does not require code in kolla-kubernetes. | 18:00 |
| rhallisey | what | 18:00 |
| kfox1111 | thats... unusual. | 18:00 |
| kfox1111 | inc0: we're using native a bit. having done many an upgrade, I don't think its going to be possible for a while. | 18:01 |
| kfox1111 | or else, we'd be implementing a workflow engine inside the containers, instead of using an existing one outside. | 18:01 |
| inc0 | yeah upgrades with native will be hard | 18:02 |
| sdake | *3* months in guys | 18:02 |
| sdake | dont solve world hunger, solve one problem at a time :) | 18:02 |
| *** neilus has quit IRC | 18:02 | |
| *** mewald1 has quit IRC | 18:02 | |
| harlowja | is the whole kolla/ansible stuff still splitting off? | 18:02 |
| sdake | harlowja roger | 18:03 |
| sdake | same goernance | 18:03 |
| sdake | different repos | 18:03 |
| harlowja | kk | 18:03 |
| kfox1111 | yes. | 18:03 |
| inc0 | harlowja, yeah, but we're discussing on keeping containers in kolla-core | 18:03 |
| rhallisey | kfox1111, let's stick with what we have for BPs for now | 18:03 |
| harlowja | cools | 18:03 |
| inc0 | configs* | 18:03 |
| inc0 | containers stays there;) | 18:03 |
| sdake | and we are not renaming the kolla repo :) | 18:03 |
| harlowja | i'm slowly but surely making it possible to start absorbing this stuff | 18:03 |
| inc0 | no, but I feel we need to set up nomenclature;) | 18:03 |
| inc0 | wonderful harlowja | 18:04 |
| sdake | yes - no repo renames of kolla pls | 18:04 |
| sdake | we lose all fantastic analytical data in a rename | 18:04 |
| sdake | and co's invest on analytics | 18:04 |
| *** ppalacios has joined #openstack-kolla | 18:05 | |
| sdake | ok so is that settled then? | 18:05 |
| rhallisey | ya | 18:05 |
| rhallisey | summit | 18:05 |
| kfox1111 | tabled till the summit. | 18:05 |
| kfox1111 | workflow lands in kolla-kubernetes until then. | 18:06 |
| sdake | cool | 18:06 |
| sdake | ok now rc2 ;) | 18:06 |
| *** david-lyle has joined #openstack-kolla | 18:06 | |
| sdake | rhallisey kfox1111 we need to tag kolla-kubernetes prior to summit as well | 18:06 |
| kfox1111 | why? | 18:07 |
| rhallisey | sdake, tag but no stable release right? | 18:07 |
| sdake | we get kicked out of the release as an entire proejct if our deliverables don't "deliver" | 18:07 |
| sdake | rhallisey right just a tag | 18:07 |
| rhallisey | kk | 18:07 |
| kfox1111 | ah. | 18:07 |
| kfox1111 | k. | 18:07 |
| sdake | branches happen onx releases | 18:07 |
| *** ldeptula has quit IRC | 18:08 | |
| kfox1111 | I thought kolla-kubernetes was release independent though | 18:08 |
| sdake | kfox1111 expectations for release indepndent projects is to release on mielstones | 18:08 |
| inc0 | kolla-k8s is | 18:08 |
| inc0 | kolla is not | 18:08 |
| sdake | kollais release trailing | 18:08 |
| *** HyperJohnGraham has joined #openstack-kolla | 18:08 | |
| kfox1111 | sdake: ah. | 18:08 |
| sdake | did i mention i hate the dentist ;) | 18:08 |
| inc0 | sbezverk_, kfox1111, what was command to release lock in ceph | 18:09 |
| inc0 | ? | 18:09 |
| kfox1111 | rbd lock somethingorother. | 18:09 |
| kfox1111 | rbd --help | 18:09 |
| kfox1111 | its like an rbd lock ls first to get the host/token and then something like rbd unlock host token or something. | 18:10 |
| inc0 | yeah so --shared didn't help:/ | 18:12 |
| kfox1111 | it onlys works with ro volumes | 18:12 |
| *** eaguilar has joined #openstack-kolla | 18:13 | |
| inc0 | I'm half tempted to have a loop while true remove locks | 18:14 |
| inc0 | for demo | 18:14 |
| kfox1111 | inc0: with the ceph patch about to merge, there is only two things using rbd that will be affected by this issue. | 18:14 |
| *** shardy is now known as shardy_afk | 18:14 | |
| kfox1111 | rabbit and mariadb. | 18:14 |
| inc0 | 2 things we care about most | 18:14 |
| kfox1111 | rabbit doesn't actuallly require persistance. | 18:14 |
| kfox1111 | we could disable it. | 18:14 |
| inc0 | we show ha | 18:14 |
| inc0 | let's do ha | 18:15 |
| kfox1111 | rabbit will failover to another node quickly, then the rpc's will timeout and resend. | 18:15 |
| inc0 | if we solve it for mariadb we solve it for rabbit too | 18:15 |
| kfox1111 | mariadb's another story. | 18:15 |
| kfox1111 | that one must be persistant. | 18:15 |
| inc0 | story we need to solve | 18:15 |
| kfox1111 | we s hould just ha it. | 18:15 |
| kfox1111 | should be doable. | 18:15 |
| kfox1111 | brb | 18:15 |
| inc0 | and do non-floating mariadb?:) | 18:15 |
| sdake | it would be nice if you could show workflow | 18:16 |
| sdake | but i know thats a stretch | 18:16 |
| inc0 | we might just as well deploy mariadb with kolla-ansible | 18:16 |
| inc0 | sdake, that's a goal | 18:16 |
| sdake | inc0 nah we want to show a running openstack on kuberenetes | 18:16 |
| sdake | mariadb deployed with kolla-ansible doesn't show that | 18:16 |
| inc0 | sdake, we are talking about different issue | 18:17 |
| sdake | cool | 18:17 |
| inc0 | sbezverk_, what are things k8s folk suggested? | 18:17 |
| *** HyperJohnGraham has quit IRC | 18:18 | |
| sdake | 2 weeks to sort out what kolla-kubernetes demo is | 18:18 |
| kfox1111 | mariadb could float, a node failure though wouldn't fully autorecover yet, but it wouldn't matter. | 18:18 |
| *** dwalsh has quit IRC | 18:18 | |
| *** schwicht has joined #openstack-kolla | 18:19 | |
| kfox1111 | the other option, if gallera doesn't like the floating ip's, | 18:19 |
| kfox1111 | is we could just usae a daemonset for now. | 18:20 |
| *** rstarmer has joined #openstack-kolla | 18:20 | |
| sdake | kfox1111 - ntoice the message int his bug re /sys | 18:20 |
| kfox1111 | I think we want to have that option in the toolbox anyway, as some ops will ike that more. | 18:20 |
| sdake | https://review.openstack.org/#/c/381942/2 | 18:20 |
| sdake | kfox1111 rather https://bugs.launchpad.net/bugs/1631072 | 18:21 |
| openstack | Launchpad bug 1631072 in kolla "iscsid container: mkdir /sys/kernel/config: operation not permitted" [Undecided,New] | 18:21 |
| kfox1111 | sdake: I found the issue on mine. it was a config option not making it into a template the way it shoudl have. | 18:21 |
| sdake | kfox1111 roger | 18:22 |
| *** mkoderer has quit IRC | 18:22 | |
| kfox1111 | I'm guessing that one is an issue with lack of privileged flag. | 18:22 |
| inc0 | btw I have idea for locking issue | 18:23 |
| inc0 | . | 18:23 |
| inc0 | we remove lock on container start | 18:24 |
| inc0 | lock is added before container is started, it spawns, we run additional script in kolla-start to remove lock | 18:24 |
| inc0 | dirty but works | 18:25 |
| inc0 | can we do this? | 18:25 |
| kfox1111 | it won't start. the volume is moutned before any user code. | 18:25 |
| inc0 | yes, but: | 18:26 |
| inc0 | 1. we do bootstrap - volume monuted, lock->1 | 18:26 |
| inc0 | 2. we remove lock->0 | 18:26 |
| *** tonanhngo has joined #openstack-kolla | 18:26 | |
| sbezverk_ | inc0: sorry was on the meeting | 18:26 |
| inc0 | 3. we run actual bootsrap, no lock present | 18:26 |
| kfox1111 | oh. you mean remove the lock while its still mounted? | 18:26 |
| inc0 | 4. k8s run mariadb lock->1 | 18:26 |
| inc0 | yup | 18:27 |
| sbezverk_ | inc0: they suggested to tweak a couple of timers | 18:27 |
| kfox1111 | might work... scary as hell. :) | 18:27 |
| inc0 | works, just tested it | 18:27 |
| sbezverk_ | which improved things a bit | 18:27 |
| wirehead_ | heh | 18:27 |
| inc0 | kfox1111, yup, scary as hell, but we need to code in support for fencing in k8s | 18:27 |
| inc0 | anyway | 18:27 |
| kfox1111 | still. would prefer clustered mariadb to solve. no one would wnat to run the lockless way in production. | 18:27 |
| kfox1111 | so its a jimmied up demo. | 18:27 |
| sbezverk_ | inc0: the main issue now is behavior of petsets | 18:28 |
| kfox1111 | I'd rather not be too fake in the demo. that's off putting. | 18:28 |
| inc0 | kfox1111, yeah, for demo it's enough, for prod we will figure it out | 18:28 |
| inc0 | it's not *that* fake | 18:28 |
| inc0 | and we won't hide it | 18:28 |
| rodrigo_pereira | may someone help me, I'm getting http 500 error on horizon dashboard, when running aio on vagrant environment. I'm not finding anything on logs, but i have put django on DEBUG true, I'm got this: http://pastebin.com/m8nEL4Zj | 18:28 |
| sbezverk_ | inc0: when replacation controller or deployment controller need to start new instance instead of failed one it does not wait for old instance complete termination | 18:28 |
| kfox1111 | we still have a huge moutn of awesome to show, without having to show off that. | 18:28 |
| inc0 | I mean we don't presume prod-readiness yet | 18:28 |
| sbezverk_ | which is not the case wit hpersets | 18:29 |
| kfox1111 | sbezverk_: even more unsafe. | 18:29 |
| sbezverk_ | oops petsets | 18:29 |
| inc0 | kfox1111, issue with clustered db is...it might be slower than if we just deploy it old style | 18:29 |
| kfox1111 | it shouldn't take 2 weeks to get a mariadb cluster going. | 18:29 |
| *** dwalsh has joined #openstack-kolla | 18:29 | |
| kfox1111 | inc0: so? how fast does it have to be to launch a vm or 2? | 18:29 |
| inc0 | kfox1111, what I'm saying we don't have any real value of running it on k8s without autohealinng | 18:30 |
| inc0 | and databases weren't written with "floating around" in mind | 18:30 |
| kfox1111 | it has value. one node won't ause the entire system to lock up waiting for it to come back. | 18:30 |
| sbezverk_ | inc0: it we can get recovery aournd 5 minutes, would it be acceptable for now? | 18:30 |
| inc0 | well, we need to fence it anyway | 18:30 |
| kfox1111 | and require dangerious workarounds. :/ | 18:31 |
| inc0 | kfox1111, today | 18:31 |
| inc0 | we need this feature in k8s, I'll write if I have to | 18:31 |
| inc0 | everyone needs this feature, I see some code in master already | 18:31 |
| kfox1111 | I'm saying, I think we can fix it right, for the demo. rather hten hack something together that will never fly in production. | 18:31 |
| kfox1111 | yeah. the lock/fencing issue is in progress. | 18:32 |
| kfox1111 | thye just build a new issue for it... let me get you a link. | 18:32 |
| kfox1111 | https://github.com/kubernetes/kubernetes/pull/34160 | 18:32 |
| sbezverk_ | inc0: could you answer my question? | 18:32 |
| kfox1111 | in the mean time, gallara should work. | 18:33 |
| inc0 | sbezverk_, what takes this long? that's my question;) | 18:33 |
| sbezverk_ | inc0: hm let me state it third time :-) | 18:33 |
| *** msimonin1 has joined #openstack-kolla | 18:33 | |
| inc0 | it doesn't wait for termination | 18:33 |
| inc0 | so it's not termination | 18:33 |
| sbezverk_ | it takes long because petset waits for complete termination of instance before starting new one | 18:34 |
| inc0 | even if node fails? | 18:34 |
| sbezverk_ | yes | 18:34 |
| sbezverk_ | node is already in notready state but isnt | 18:34 |
| kfox1111 | thats kind of a bug too. | 18:34 |
| sbezverk_ | instance stays in running state by default 5 minutes | 18:34 |
| *** msimonin has quit IRC | 18:34 | |
| inc0 | so there is some timeout involved for it to go down? | 18:34 |
| sbezverk_ | with new timers I managed to reduce it | 18:35 |
| inc0 | can we make it 30s? | 18:35 |
| kfox1111 | sbezverk_: while a bit of a hack, much more comfortable with this idea... | 18:35 |
| sbezverk_ | but then petset container get stuck in terminating state | 18:35 |
| kfox1111 | why not make a script that polls for dead nodes, and kills pods on them? | 18:35 |
| kfox1111 | it can run once a second and delete the pod with --now ? | 18:35 |
| inc0 | kfox1111, hard to kill a dead node;) | 18:35 |
| sbezverk_ | inc0: done, but this timers do not change "terminating" part :-( | 18:35 |
| kfox1111 | no the node, the petset pod. | 18:36 |
| inc0 | kfox1111, if node is dead, petset pod is dead too | 18:36 |
| sdake | rbergeron ping | 18:36 |
| inc0 | just k8s doesn't knwo about it | 18:36 |
| inc0 | ehh I like this project less and less;) | 18:36 |
| sbezverk_ | inc0: it just a figment of memory of kube brain not letting it go ;-) | 18:36 |
| kfox1111 | inc0: its fine. your just wanting it all now. ;) | 18:36 |
| sdake | welcome to distributed systems madness | 18:37 |
| kfox1111 | it'll get there. | 18:37 |
| inc0 | can we splatter this brain on the wall and make it go faster?;) | 18:37 |
| inc0 | kfox1111, I know | 18:37 |
| sbezverk_ | inc0: I filed an issue with kube this morning, so far no answer | 18:37 |
| inc0 | just listing in my mind stuff to look at when I'd like to contribute to it | 18:37 |
| kfox1111 | sbezverk_: what do you think? if we detect nodeDown state, will deleting the pod with --now clear it right away? | 18:38 |
| *** srwilkers has quit IRC | 18:38 | |
| inc0 | that could be part of our fencing script | 18:38 |
| sbezverk_ | kfox1111: do not think so, as when you run kuebctl delete on Terminating pod it just tells you that the pod is already deleting | 18:38 |
| inc0 | node down -> kill all pods -> fence the crap out of ceph -> release lock -> restart | 18:39 |
| kfox1111 | sbezverk_: --now does somethign different though. | 18:39 |
| sbezverk_ | will try it next time I kill my compute node | 18:39 |
| inc0 | we can make shellscript runing in look for it | 18:39 |
| kfox1111 | sbezverk_: --now tells k8s to ignore the term/prestop hooks and hard kill the pod. | 18:39 |
| inc0 | and say on demo "yeah, kube needs work | 18:40 |
| inc0 | and say on demo "this is workaround, kube needs work, help needed" | 18:40 |
| kfox1111 | I've had timeouts I've set on pods that were set to weeks, and the orchestration scripts failed to work, and killing them with --now worked. | 18:40 |
| sbezverk_ | :-) | 18:40 |
| * kfox1111 nods. | 18:40 | |
| sbezverk_ | kfox1111: are you sure they were in Terminating state? | 18:40 |
| kfox1111 | sbezverk_: yup. | 18:40 |
| kfox1111 | the only difference was the node was still up. so havent tried that part. | 18:41 |
| inc0 | also ad removing lock is bad | 18:41 |
| kfox1111 | but --now does cause it to ignore timeout on most cases. so it might on this case too. | 18:41 |
| inc0 | it really isn't, not fencing node is bad | 18:41 |
| sbezverk_ | kfox1111: that might change a thing ot two ;-) | 18:41 |
| inc0 | but having lock erected for failed node - no bueno too | 18:41 |
| kfox1111 | inc0: is there a way to fence a node in ceph? | 18:41 |
| inc0 | yes | 18:41 |
| kfox1111 | ok. so, how about this: | 18:41 |
| kfox1111 | while true; | 18:42 |
| inc0 | http://docs.ceph.com/docs/jewel/cephfs/eviction/ | 18:42 |
| MarMat | ping Jeffrey4l_ | 18:42 |
| kfox1111 | kubectl get nodes | grep down > names | 18:42 |
| kfox1111 | if names |wc -l > 0 | for each node | 18:42 |
| kfox1111 | ceph fence node | 18:42 |
| kfox1111 | ceph remove all locks on node | 18:42 |
| inc0 | ahh this is for fs | 18:42 |
| kfox1111 | kubectl delete pods on node --now | 18:42 |
| inc0 | hmm, let see if it works for rbd | 18:42 |
| kfox1111 | sleep 1 | 18:42 |
| kfox1111 | done | 18:42 |
| sbezverk_ | kfox1111: that might solve only locking problem | 18:43 |
| sbezverk_ | terminating mariadb container is still unsolved problem | 18:43 |
| inc0 | kfox1111, so we can evict with cephfs | 18:44 |
| *** haplo37_ has quit IRC | 18:45 | |
| *** haplo37_ has joined #openstack-kolla | 18:47 | |
| sbezverk_ | kfox1111: just tried and it worked with --now | 18:47 |
| inc0 | kfox1111, can we move to cephfs driver quickly? | 18:49 |
| *** harlowja has quit IRC | 18:50 | |
| *** Pavo has quit IRC | 18:51 | |
| kfox1111 | sbezverk_: ok, so that script above might just do the trick? | 18:51 |
| kfox1111 | inc0: I think thats a tradeoff.... | 18:52 |
| kfox1111 | cephfs only recommends having one master. | 18:52 |
| kfox1111 | so we tradeoff cephfs mds for mariadb. | 18:52 |
| sbezverk_ | kfox1111: yes, for now I can even start manually to clean up all petsets containers | 18:52 |
| inc0 | hmm | 18:52 |
| sbezverk_ | kfox1111: never heard before what is cephfs? | 18:53 |
| kfox1111 | sbezverk_: k. I vote for a script like that. its not very hacky and its stuff that will all go into k8s when ready. | 18:53 |
| inc0 | so we need to figure out ceph fence node part for rbd | 18:53 |
| kfox1111 | cephfs is a distributed filesystem built on top of the ceph object store. | 18:53 |
| sbezverk_ | kfox1111: have nothing against script ideally would be nice to hook it somewhere ;-) | 18:53 |
| kfox1111 | inc0: yeah. if we can solve that, the script should be easy to flesh out with the remaining details. | 18:54 |
| inc0 | problem is it's actually going to be hard with ceph arch | 18:54 |
| inc0 | we'd need to do iptables drop on all the osds;) | 18:54 |
| inc0 | that has objects for this particular image | 18:55 |
| inc0 | well..pgs.. | 18:55 |
| inc0 | for pool...you get my drift | 18:55 |
| inc0 | but that's true for every shared storage we ever want to have if we want it to float around | 18:55 |
| *** Pavo has joined #openstack-kolla | 18:56 | |
| *** david-lyle has quit IRC | 18:57 | |
| inc0 | one idea | 18:57 |
| inc0 | make a rbd ro -> create cow snapshot -> mount this snapshot to new pod | 18:57 |
| inc0 | flatten when safe | 18:57 |
| *** neilus has joined #openstack-kolla | 19:00 | |
| inc0 | on the bright note sbezverk_ wirehead_ rhallisey we'll going to have stack of NUCs for talk:) | 19:02 |
| inc0 | 4 of them | 19:02 |
| kfox1111 | oh. so there isn't a ceph fence after all? | 19:02 |
| inc0 | not for rbd | 19:02 |
| inc0 | my bad | 19:02 |
| kfox1111 | what about the object level? rbd just sits on top. | 19:02 |
| kfox1111 | or were yiou thinking cephfs? | 19:02 |
| *** huikang_ has quit IRC | 19:03 | |
| inc0 | cephfs has fencing - confirmed | 19:03 |
| *** huikang has joined #openstack-kolla | 19:03 | |
| kfox1111 | ok... one other hack I can think of... not sure if its a good idea or not. | 19:03 |
| kfox1111 | well... I guess that doesn't matter. | 19:04 |
| kfox1111 | if we know the node is dead, we can just pull the lock. | 19:04 |
| inc0 | we need to fence it | 19:04 |
| inc0 | for prod | 19:04 |
| kfox1111 | ok.... what if we just fence ourself? | 19:04 |
| sbezverk_ | kfox1111: if we could somehow get notification from kubernetes event and run a script on this event that would be a good solution | 19:04 |
| kfox1111 | take the loop above, | 19:04 |
| kfox1111 | and change ceph fence to "ipmi power off node" | 19:05 |
| inc0 | yeah well, that works | 19:05 |
| kfox1111 | the script would just loop until k8s finds a dead node, and we fence it by forcing it off, then remove the lock and killing existing pods? | 19:05 |
| inc0 | if we could make it purely on ceph level we can use k8s for it | 19:05 |
| *** neilus has quit IRC | 19:05 | |
| rhallisey | inc0, :) | 19:05 |
| kfox1111 | the script could run in a pod in k8s... :) | 19:06 |
| kfox1111 | thats the k8s way. :) | 19:06 |
| inc0 | :D | 19:06 |
| inc0 | and it's really stateless | 19:06 |
| inc0 | so no need for petset | 19:06 |
| kfox1111 | and you could run 2 so one node faulure doesn't kill it. | 19:06 |
| inc0 | we can even have couple replicas of it:) | 19:06 |
| kfox1111 | yup. | 19:06 |
| inc0 | I think if we could pull that off with just ceph that would be awesome, but I guess that's good science project for later:) | 19:07 |
| inc0 | bottom line, we want stuff to float around -> we need a way to fence them | 19:07 |
| *** huikang has quit IRC | 19:08 | |
| *** absubram has joined #openstack-kolla | 19:08 | |
| inc0 | kfox1111, one way to do ceph fencing | 19:09 |
| inc0 | you have mariadb object | 19:09 |
| inc0 | mariadb obj -> snap! -> new pod gets to write cow object made on top of snap! -> remove original mariadb obj | 19:10 |
| *** mgoddard has joined #openstack-kolla | 19:10 | |
| *** mgoddard_ has quit IRC | 19:10 | |
| inc0 | old pod dies in fire as we just removed it's disk:( | 19:10 |
| inc0 | what do you think? | 19:11 |
| inc0 | nope, we need to stop io to it before:S | 19:12 |
| *** salv-orlando has quit IRC | 19:13 | |
| kfox1111 | yeah. | 19:13 |
| kfox1111 | I think just forcing an extra power off should let us unlock safely. | 19:13 |
| inc0 | yeah I guess | 19:13 |
| inc0 | ipmi poweroff rules them all | 19:13 |
| kfox1111 | its a pretty common pattern too. | 19:13 |
| kfox1111 | ops are use to that. | 19:13 |
| inc0 | I know | 19:13 |
| kfox1111 | you could even use the redhat fencing tools to do it. so it would be even more familior | 19:14 |
| kfox1111 | the k8s folks were talking about doing the same. | 19:14 |
| inc0 | ok, ipmi poweroff it is | 19:14 |
| inc0 | until kube grows a fencing | 19:14 |
| kfox1111 | this is something we probably can ship too, in production kolla-kubernetes until k8s grows it. | 19:15 |
| inc0 | fencing pod?;) | 19:15 |
| inc0 | yeah I guess | 19:15 |
| *** mgoddard has quit IRC | 19:15 | |
| inc0 | it isn't *that* ugly | 19:16 |
| inc0 | and will bring down db downtime on failure to seconds | 19:16 |
| *** coolsvap has quit IRC | 19:16 | |
| sbezverk_ | how do you clear ceph locks from the fencing pod? | 19:17 |
| sbezverk_ | ssh to ceph? | 19:17 |
| kfox1111 | we have a ceph-admin pod in the ps I'm working on. | 19:17 |
| inc0 | nah, we do container with ceph-common package and keyring | 19:17 |
| kfox1111 | if the fencing pod has a service acount (should), | 19:17 |
| kfox1111 | it can kubectl exec ceph-common rbd .... | 19:18 |
| kfox1111 | it can kubectl exec ceph-admin rbd .... | 19:18 |
| kfox1111 | ceph-admin already has an admin key | 19:18 |
| inc0 | we can have k8s-toolbox:) | 19:18 |
| inc0 | with ceph-admin and our fencing stuff in same container | 19:19 |
| sbezverk_ | kfox1111 if poeple do not use ceph-admin, maybe ceph-common in this pod is good alternatives | 19:19 |
| kfox1111 | hmm... the admin pod's not garanteed to be up though, as it could be on the fenced node. | 19:19 |
| *** daneyon has joined #openstack-kolla | 19:19 | |
| inc0 | kfox1111, can we made it into multiple replicas and rc it? | 19:19 |
| inc0 | it doesn't run anything right? | 19:19 |
| sbezverk_ | inc0 ceph-amin is not mandatory | 19:19 |
| kfox1111 | ok. maybe best to mount the ceph-kolla secret into the fence pod then. | 19:19 |
| inc0 | sbezverk_, container, not package | 19:20 |
| sbezverk_ | kfox1111 +1 | 19:20 |
| kfox1111 | that keeps it nice and simple. | 19:20 |
| inc0 | yeah | 19:20 |
| sbezverk_ | agree mounting secret to fencing sounds very kubernetes like ;-) | 19:20 |
| inc0 | we'd need ipmi conf and secrets in it too | 19:21 |
| inc0 | so we need new image | 19:21 |
| inc0 | let's publish it to kolla after we branch out neutron | 19:21 |
| sbezverk_ | inc0: so we will need new docker in kolla | 19:21 |
| sdake | one thing i'd ask | 19:21 |
| kfox1111 | yeah. new image with ceph bits, and redhat fencing tools. | 19:21 |
| sdake | is make sure anything you demo is in the queue | 19:21 |
| kfox1111 | the rest we can pull in from configmaps/secrets. | 19:21 |
| inc0 | sdake, sure | 19:22 |
| sdake | i know it can't be in the repo because the repo is frozen | 19:22 |
| inc0 | for one more week | 19:22 |
| inc0 | we can just not merge it till then | 19:22 |
| sdake | right | 19:22 |
| kfox1111 | +! | 19:22 |
| kfox1111 | +1 | 19:22 |
| inc0 | it's pretty elegant imho | 19:23 |
| inc0 | at east not too horrible | 19:23 |
| *** dwalsh has quit IRC | 19:23 | |
| kfox1111 | so close... got everything working in the gate, including launching vm's, setting up networking, pinging between vm's. | 19:23 |
| kfox1111 | just gota fix cinder, and I think we're done. :) | 19:23 |
| inc0 | yay | 19:23 |
| inc0 | this gate will be better than kolla's;) | 19:24 |
| *** daneyon has quit IRC | 19:24 | |
| sdake | cool gate parity ;-) | 19:24 |
| inc0 | no, kolla-ansible will actually be slightly worse;) | 19:24 |
| sbezverk_ | kfox1111: if you are stll busy with gate, I can start fencing pod | 19:24 |
| inc0 | we need to install ceph and cinder in kolla-ansible gate too | 19:24 |
| inc0 | sbezverk_, let me please;) | 19:24 |
| inc0 | or together | 19:24 |
| inc0 | I want to have at least one patchset in kolla-k8s before summit :( | 19:25 |
| kfox1111 | sbezverk_: yeah, please do. maybe base it on the oustanding gate ps though. | 19:25 |
| sbezverk_ | inc0: sure as long as it does not get dealyed by other of your responsibilities ;-) | 19:25 |
| kfox1111 | it would be nice to use the ceph-kolla secret it sets up. | 19:25 |
| kfox1111 | as its restricted to the rbd volumes. | 19:25 |
| inc0 | kfox1111, it sets this secret in gates tho right | 19:26 |
| inc0 | ? | 19:26 |
| inc0 | we don't have secret_generator for it | 19:26 |
| kfox1111 | yeah. it sets up secrets for it. | 19:26 |
| inc0 | as we don't have ceph keyrings in passwords.yml | 19:26 |
| kfox1111 | right. | 19:26 |
| kfox1111 | its in the setup_gate.sh for now. | 19:26 |
| kfox1111 | using the ceph-admin stuff it sets up. | 19:27 |
| *** dwalsh has joined #openstack-kolla | 19:27 | |
| inc0 | yeah, we won't require setup gate, just kubectl create | 19:27 |
| sbezverk_ | we need to require ceph-secret to be created in either cases | 19:27 |
| kfox1111 | after this ps merges, | 19:27 |
| inc0 | we need both ceph auth creds and ipmi auth creds there | 19:27 |
| kfox1111 | I'm going to break the setup_gate into a few seperate scripts, | 19:27 |
| kfox1111 | making it easy to run a few of the bits. | 19:27 |
| sbezverk_ | even if ceph-admin pod is not used | 19:27 |
| kfox1111 | like setting up the ceph-kolla secret, make the rbd voluems, etc. | 19:28 |
| inc0 | I think I'll rather add tpls for these resources to kolla-k8s and require them in config | 19:28 |
| kfox1111 | so we can easily call it outside the gate, until the workflow catchs up. | 19:28 |
| kfox1111 | its a secret generation step, its gota go through a ceph client. | 19:28 |
| kfox1111 | then the output of that is put in a template. | 19:28 |
| inc0 | actually no, you need keys | 19:29 |
| sbezverk_ | kfox1111: please without any dependencies :-) if I create ceph-secret manually, nothing should prevent me from using in fencing pod | 19:29 |
| kfox1111 | yeah. that. | 19:29 |
| inc0 | you need key to get a key:) | 19:29 |
| inc0 | so we can just require *the key*:) | 19:29 |
| kfox1111 | sbezverk_: you can upload ceph-secret as ceph-client-admin-key and lauch ceph-admin | 19:29 |
| *** dwalsh has quit IRC | 19:29 | |
| kfox1111 | and do all the rest of the key creations / rbd volume and pool setup. | 19:30 |
| sbezverk_ | kfox1111: my point is ceph-admin is nice to have but not mandatory and it should stay like that I think | 19:30 |
| inc0 | yeah, I think fence pod should be self-sufficient | 19:30 |
| kfox1111 | sbezverk_: I agree, but then you have to manually do a lot of crap in ceph to actualy support ceph. | 19:30 |
| kfox1111 | using an admin key for it all is really bad practice. | 19:30 |
| sbezverk_ | kfox1111: and it is my choise | 19:31 |
| inc0 | kfox1111, but we don't need to use admin key | 19:31 |
| kfox1111 | sure. thats fine. but don't recommend it to anyone else. | 19:31 |
| inc0 | that doesn't mean we need to create ceph user | 19:31 |
| sbezverk_ | cool | 19:31 |
| kfox1111 | inc0: sure. you can manulaly create all the stuff through some other means. | 19:31 |
| inc0 | if somebody wants to use admin- fine, but let operator just provide us with user/key | 19:31 |
| kfox1111 | inc0: I provided the bits so you can use them if you'd like. | 19:31 |
| *** dwalsh has joined #openstack-kolla | 19:31 | |
| inc0 | we require user/key for ceph and user/pass for ipmi | 19:32 |
| inc0 | and do kubectl create based on this | 19:32 |
| kfox1111 | and I'm going to break out the script bits so you can run that little bit of the worfklow if you'd like. | 19:32 |
| kfox1111 | inc0: right. | 19:32 |
| inc0 | do we have template for secret anywhere | 19:32 |
| inc0 | ? | 19:32 |
| kfox1111 | what are you trying to do? | 19:33 |
| sbezverk_ | inc0: the other day I sent you paste with secret template we used for ceph | 19:33 |
| inc0 | so currently we have key for ceph in /etc/kolla-kubernetes/kolla-kubernetes.yml | 19:33 |
| inc0 | sbezverk_, I know, but do we have it in tree?:) | 19:34 |
| inc0 | we need 2 more secrets that won't be in original passwords.yml | 19:34 |
| kfox1111 | inc0: usually you create secrets with kubectl secret create | 19:34 |
| inc0 | ceph secret and ipmi secret | 19:34 |
| kfox1111 | inc0: heres what I'm doing: | 19:34 |
| kfox1111 | https://review.openstack.org/#/c/381041/67/tools/setup_gate.sh | 19:34 |
| sbezverk_ | no we do not because we require operator to create them before the installation of kolla | 19:34 |
| kfox1111 | see starting about line 389, sets up the ceph bits. | 19:35 |
| *** huikang has joined #openstack-kolla | 19:35 | |
| inc0 | sbezverk_, I see...I guess we can just require them created and handle it in workflow | 19:35 |
| *** janem has quit IRC | 19:36 | |
| inc0 | bbiaf, moving to home with this, I need bigger screens ;) | 19:36 |
| sbezverk_ | inc0: yep | 19:36 |
| *** eaguilar has quit IRC | 19:36 | |
| sbezverk_ | unless of course they will use Kevin's bits | 19:37 |
| sbezverk_ | to automate it | 19:37 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 19:37 |
| kfox1111 | for tipical setup, we will need 4 ceph keys. one for rbd, one for nova, one for glance, one for cinder. | 19:39 |
| kfox1111 | and if you want to use the workflow to set it up, the admin key just long enough to create the other keys. | 19:39 |
| sbezverk_ | kfox1111: I guess that is the difference between me running cluster in the lab with one key to rule them all and a key per hobbit ;-) in real operation | 19:40 |
| * kfox1111 nods. | 19:41 | |
| kfox1111 | though, a word of advice that a bunch of devs miss. if your not testing like your users actually will use it, then you run the risk of not producing useful stuff. | 19:42 |
| sbezverk_ | sure | 19:42 |
| sbezverk_ | make sense | 19:42 |
| kfox1111 | shortcuts are fine for development to speed development, so long as the stuff users actually will use is still tested somehow. | 19:43 |
| kfox1111 | and the shortcuts don't cause too much extra development fixing stuff that breaks on that side. | 19:43 |
| *** inc0 has quit IRC | 19:45 | |
| kfox1111 | using stuff like admin keys or root for development can hide problems and make it really hard to debug later. you suddently see a huge pile of issues when you turn off root/adminness if you don't test all the time without it. | 19:45 |
| kfox1111 | arg... the queue seems to be plugged. | 19:48 |
| *** dave-mccowan has quit IRC | 19:49 | |
| sbezverk_ | kfox1111: I guess poeple tried to test last bits for newton before pushing it out today | 19:51 |
| mliima | bye guys, :) | 19:52 |
| *** mliima has quit IRC | 19:52 | |
| *** dave-mccowan has joined #openstack-kolla | 19:58 | |
| *** msimonin1 has quit IRC | 20:00 | |
| *** msimonin has joined #openstack-kolla | 20:01 | |
| *** dave-mcc_ has joined #openstack-kolla | 20:01 | |
| *** haplo37 has quit IRC | 20:02 | |
| *** dave-mccowan has quit IRC | 20:04 | |
| *** g3ek has quit IRC | 20:06 | |
| *** david-lyle has joined #openstack-kolla | 20:06 | |
| *** inc0 has joined #openstack-kolla | 20:07 | |
| *** g3ek has joined #openstack-kolla | 20:07 | |
| *** inc0_ has joined #openstack-kolla | 20:07 | |
| *** inc0_ has quit IRC | 20:08 | |
| inc0 | back | 20:08 |
| *** sdake has quit IRC | 20:09 | |
| *** dwalsh has quit IRC | 20:09 | |
| *** haplo37 has joined #openstack-kolla | 20:11 | |
| *** lrensing has quit IRC | 20:14 | |
| inc0 | oh we already have ceph secret template:) | 20:16 |
| sbezverk_ | meeting time, bbl | 20:16 |
| *** harlowja has joined #openstack-kolla | 20:17 | |
| inc0 | harlowja, when do you think you guys could start moving towards Kolla? | 20:17 |
| harlowja | inc0 soooooo | 20:17 |
| harlowja | i'm trying to get through a state of having a CI that can actually run reliably first | 20:17 |
| harlowja | (pre and post patch) | 20:18 |
| harlowja | we don't have that; so that's my #0 step | 20:18 |
| harlowja | (basic things ...) | 20:18 |
| inc0 | you mean internal CI or openstack one? | 20:18 |
| harlowja | internal | 20:18 |
| harlowja | i'd like for that internal CI to be able to use kolla but not sure how that can happen yet | 20:18 |
| harlowja | its jenkin based | 20:18 |
| harlowja | and i didn't think kolla had a 'unittest' kind of task? | 20:19 |
| harlowja | where i can test before and after patching | 20:19 |
| inc0 | well if you plan to CI build, that's gonna be tricky | 20:19 |
| inc0 | but I'm not sure if you have to tbh | 20:19 |
| harlowja | if we didn't have patches | 20:19 |
| harlowja | since we do u sorta need it | 20:19 |
| inc0 | ok, fair enough, you do know how to customize dockerfile right? | 20:20 |
| harlowja | yes, is there any way to tell it to apply some patches? | 20:20 |
| inc0 | sure, hold on | 20:20 |
| harlowja | without modifying kolla docker files :-P | 20:20 |
| harlowja | cause i don't want more patches | 20:20 |
| kfox1111 | harlowja: see this: | 20:21 |
| inc0 | https://github.com/openstack/kolla/blob/master/doc/image-building.rst#dockerfile-customisation | 20:21 |
| kfox1111 | https://review.openstack.org/#/c/381041/68 | 20:21 |
| *** fguillot has quit IRC | 20:21 | |
| kfox1111 | its close to merging, and is a gate job for standing it all up. | 20:21 |
| inc0 | kfox1111, we're not talking about kolla-k8s now;) | 20:21 |
| *** dwalsh has joined #openstack-kolla | 20:21 | |
| kfox1111 | oh. ok. sry. | 20:22 |
| inc0 | harlowja, btw when you guys feel like it, k8s impl should be waiting:) | 20:22 |
| harlowja | kk | 20:24 |
| harlowja | kfox1111 perhaps someday :-P | 20:24 |
| harlowja | little steps here, ha | 20:24 |
| harlowja | can only handle a little bit :) | 20:24 |
| harlowja | so right now where we at is getting those post-patch unittests working btw | 20:24 |
| harlowja | they weren't all passing | 20:24 |
| harlowja | (using venvs) | 20:25 |
| harlowja | once those work i'd be fine with doing the docker file customization | 20:25 |
| *** gfidente has quit IRC | 20:25 | |
| inc0 | you can either inject some random code into certain places (we can grow more places like that as needed:)) | 20:26 |
| inc0 | modify packages, both pip and apt | 20:26 |
| inc0 | (add, remove, total override) | 20:26 |
| *** salv-orlando has joined #openstack-kolla | 20:27 | |
| inc0 | modify repo was there even in Mitaka | 20:27 |
| inc0 | so yeah, you can do pretty much whatever you want without touching kolla code | 20:27 |
| *** vhosakot_ has quit IRC | 20:30 | |
| *** vhosakot has joined #openstack-kolla | 20:31 | |
| harlowja | kk | 20:31 |
| harlowja | bb, in meetin | 20:31 |
| harlowja | will get the guy helping with CI in here soon :) | 20:31 |
| harlowja | cause we are nearly at post-patch stuff working (like it should be) | 20:31 |
| harlowja | and then decision around 'new container' format | 20:31 |
| harlowja | (not rpms) | 20:32 |
| *** dwalsh has quit IRC | 20:33 | |
| *** gfidente has joined #openstack-kolla | 20:39 | |
| *** gfidente has quit IRC | 20:39 | |
| *** gfidente has joined #openstack-kolla | 20:39 | |
| *** neilus has joined #openstack-kolla | 20:41 | |
| *** dave-mcc_ has quit IRC | 20:41 | |
| *** neilus has quit IRC | 20:46 | |
| *** MarMat has quit IRC | 20:47 | |
| *** Pavo has quit IRC | 20:51 | |
| otavio | Is the master branch images working? Here it fail building the base image | 20:54 |
| *** Pavo has joined #openstack-kolla | 20:55 | |
| *** vhosakot has quit IRC | 20:59 | |
| *** dave-mccowan has joined #openstack-kolla | 21:02 | |
| *** janem has joined #openstack-kolla | 21:02 | |
| *** ayoung has quit IRC | 21:02 | |
| *** huikang has quit IRC | 21:03 | |
| *** jtriley has quit IRC | 21:03 | |
| *** huikang has joined #openstack-kolla | 21:03 | |
| *** shardy_afk is now known as shardy | 21:05 | |
| *** huikang_ has joined #openstack-kolla | 21:06 | |
| *** huikang_ has quit IRC | 21:06 | |
| *** huikang_ has joined #openstack-kolla | 21:07 | |
| *** huikang has quit IRC | 21:08 | |
| *** huikang_ has quit IRC | 21:11 | |
| *** harlowja has quit IRC | 21:17 | |
| *** michauds has joined #openstack-kolla | 21:18 | |
| *** ppalacios has quit IRC | 21:22 | |
| *** shardy has quit IRC | 21:22 | |
| *** gfhellma has joined #openstack-kolla | 21:26 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 21:28 |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla-kubernetes: Workflow Gate Testing https://review.openstack.org/382021 | 21:30 |
| *** schwicht has quit IRC | 21:31 | |
| *** david-lyle_ has joined #openstack-kolla | 21:34 | |
| *** david-lyle has quit IRC | 21:34 | |
| *** david-lyle has joined #openstack-kolla | 21:35 | |
| *** david-lyle has quit IRC | 21:35 | |
| *** david-lyle has joined #openstack-kolla | 21:35 | |
| *** harlowja has joined #openstack-kolla | 21:39 | |
| *** harlowja has quit IRC | 21:39 | |
| *** david-lyle_ has quit IRC | 21:39 | |
| *** rstarmer has quit IRC | 21:39 | |
| *** harlowja has joined #openstack-kolla | 21:39 | |
| *** tonanhngo has quit IRC | 21:49 | |
| *** jheroux has quit IRC | 21:49 | |
| *** tonanhngo has joined #openstack-kolla | 21:51 | |
| *** neilus has joined #openstack-kolla | 21:54 | |
| *** david-lyle has quit IRC | 21:56 | |
| *** tonanhngo has quit IRC | 21:56 | |
| *** tonanhngo has joined #openstack-kolla | 21:57 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 21:58 |
| *** neilus has quit IRC | 21:59 | |
| harlowja | inc0 so after CI going somewhat ok | 22:00 |
| harlowja | then ya, docker stuff becomes the next question | 22:00 |
| harlowja | and then figuring out what to do with all of our puppet (that will likely fall apart right around the docker time-period) | 22:00 |
| harlowja | i'm hoping that perhaps kolla-ansible will be split off around then | 22:01 |
| harlowja | and we can use kolla-core (or whatever its called) | 22:01 |
| inc0 | kolla ansible is going to be split right after summit | 22:01 |
| harlowja | kk | 22:01 |
| *** absubram has quit IRC | 22:01 | |
| harlowja | should align good i think then | 22:01 |
| *** daneyon has joined #openstack-kolla | 22:02 | |
| harlowja | has anyone been running kolla (image building, basic testing) in jenkins ? | 22:02 |
| *** tonanhngo has quit IRC | 22:02 | |
| harlowja | or tried doing that in a automated fashion | 22:02 |
| * harlowja just curios | 22:02 | |
| harlowja | do people know if https://sdake.io/2015/11/11/the-tldr-on-immutable-infrastructure/ is still valid ? | 22:05 |
| harlowja | 'With Kolla we started with #2, briefly tried #4, and finished with technique #3.' | 22:05 |
| harlowja | is that still true? | 22:05 |
| *** daneyon has quit IRC | 22:06 | |
| kfox1111 | harlowja: I think is configurable now. | 22:09 |
| harlowja | kk | 22:09 |
| *** sdake has joined #openstack-kolla | 22:13 | |
| *** ayoung has joined #openstack-kolla | 22:21 | |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla-kubernetes: Workflow Gate Testing https://review.openstack.org/382021 | 22:22 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 22:34 |
| *** jtriley has joined #openstack-kolla | 22:37 | |
| *** rhallisey has quit IRC | 22:40 | |
| *** rhallisey has joined #openstack-kolla | 22:40 | |
| *** rhallisey has quit IRC | 22:40 | |
| *** rhallisey has joined #openstack-kolla | 22:41 | |
| *** rhallisey has quit IRC | 22:41 | |
| *** jtriley has quit IRC | 22:42 | |
| *** g3ek has quit IRC | 22:44 | |
| *** michauds has quit IRC | 22:44 | |
| *** g3ek has joined #openstack-kolla | 22:46 | |
| *** Pavo has quit IRC | 22:50 | |
| *** Pavo has joined #openstack-kolla | 22:51 | |
| *** MarMat has joined #openstack-kolla | 22:52 | |
| kfox1111 | "Horizon no longer requires Nova (or Glance) to function; it will run as long as keystone is present (for instance, swift-only deployments)." interesting. :) | 22:53 |
| *** salv-orlando has quit IRC | 23:01 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 23:05 |
| *** ayoung has quit IRC | 23:05 | |
| *** portdirect has joined #openstack-kolla | 23:12 | |
| *** gfhellma has quit IRC | 23:14 | |
| wirehead_ | It largely worked when I had it running in a Nova-and-Glance-free environment in the very early days of kube-k8s. | 23:15 |
| *** dwalsh has joined #openstack-kolla | 23:18 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 23:27 |
| kfox1111 | sbezverk_: http://docs.openstack.org/releasenotes/cinder/newton.html see the known issues section. | 23:29 |
| kfox1111 | probably will affect the iscis driver. | 23:29 |
| hogepodge | so, sorry for what will certainly be a silly question, but on a multi-node deployment how do I restart the rabbit service (containers)? | 23:30 |
| sbezverk_ | kfox1111: interesting | 23:31 |
| hogepodge | ah, nevermind, docker restart does the trick :-P | 23:38 |
| *** rodrigo_pereira has quit IRC | 23:40 | |
| harlowja | another intersting question, for questions like hogepodge is it thought that all similar types of scripts should end up as ansible in say https://github.com/openstack/kolla/tree/master/ansible/roles/rabbitmq/tasks | 23:42 |
| harlowja | like restart.yml or something should go there and ... | 23:42 |
| harlowja | as a collection of all things operators may need | 23:42 |
| hogepodge | harlowja: ah | 23:43 |
| hogepodge | yes | 23:43 |
| harlowja | seems sorta like it? i'm not sure the answer :-P | 23:43 |
| hogepodge | as it stands I have to redeploy. rabbit is sooooo picky about hostnames | 23:43 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 23:44 |
| *** dwalsh has quit IRC | 23:45 | |
| *** dwalsh has joined #openstack-kolla | 23:49 | |
| *** schwicht has joined #openstack-kolla | 23:49 | |
| *** ayoung has joined #openstack-kolla | 23:50 | |
| *** ankush has quit IRC | 23:53 | |
| kfox1111 | heh. heat finally gained conditionals. | 23:57 |
| *** ankush has joined #openstack-kolla | 23:58 | |
| harlowja | is it turing complete yet? | 23:58 |
| harlowja | thats usually when u know that <u shouldn't be using this> lol | 23:59 |
| *** dwalsh has quit IRC | 23:59 | |
| kfox1111 | no. I argued hard it should never be. | 23:59 |
| *** rbbratta has joined #openstack-kolla | 23:59 | |
| kfox1111 | :) | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!