Thursday, 2016-10-06

*** huikang has joined #openstack-kolla00:06
janemhi all, trying to enable cinder-volume. Do I have to setup vgs cinder-volumes on the docker host machine?00:12
janemAIO install00:12
janem2.0.2 Ubuntu host ubuntu-source images00:13
janemfound this: http://docs.openstack.org/developer/kolla/mitaka/cinder-guide.html00:15
janemunclear if it means on host or in cinder-volume container00:16
sbezverk_janem: on the host, container does not own any vgs00:17
janemsbezverk_: thanks!00:17
sbezverk_janem: np00:18
*** schwicht has quit IRC00:23
MarMatping Daviey00:29
*** huikang has quit IRC00:38
*** huikang has joined #openstack-kolla00:38
*** huikang_ has joined #openstack-kolla00:40
*** tonanhngo has quit IRC00:42
*** huikang has quit IRC00:43
openstackgerritDuong Ha-Quang proposed openstack/kolla: Improve VIP existence check  https://review.openstack.org/38158900:45
*** eaguilar has quit IRC00:48
*** Pavo has quit IRC00:51
*** dave-mcc_ has joined #openstack-kolla00:54
*** dave-mccowan has quit IRC00:55
*** Pavo has joined #openstack-kolla00:56
*** janem has quit IRC00:59
*** phuongnh has joined #openstack-kolla01:00
*** duonghq has joined #openstack-kolla01:00
*** HyperJohnGraham has joined #openstack-kolla01:02
HyperJohnGrahamHi All01:03
duonghqhi HyperJohnGraham01:04
*** tovin07 has joined #openstack-kolla01:08
*** tovin07 has quit IRC01:08
*** tovin07 has joined #openstack-kolla01:09
*** tovin07 has quit IRC01:09
*** tovin07 has joined #openstack-kolla01:10
*** MarMat has quit IRC01:10
*** tovin07 has quit IRC01:11
HyperJohnGrahamHi duonghq01:11
*** tovin07 has joined #openstack-kolla01:12
*** huikang_ has quit IRC01:12
HyperJohnGrahamsbezverk_: I got some new toys today for the ceph osd's http://liqid.com/liqid_powered/kingston_ep1000.html01:12
*** huikang has joined #openstack-kolla01:13
*** huikang has quit IRC01:13
*** huikang has joined #openstack-kolla01:13
*** unicell has quit IRC01:25
*** dwalsh has joined #openstack-kolla01:26
*** dave-mccowan has joined #openstack-kolla01:28
*** dave-mcc_ has quit IRC01:29
*** dwalsh has quit IRC01:31
*** dwalsh has joined #openstack-kolla01:44
*** eaguilar has joined #openstack-kolla01:44
*** MarMat has joined #openstack-kolla01:54
*** tonanhngo has joined #openstack-kolla01:55
*** dave-mccowan has quit IRC02:01
duonghqcoolsvap: did you get this: https://bugs.launchpad.net/kolla/+bug/163067802:09
openstackLaunchpad bug 1630678 in kolla "keepalived container cannot get MASTER state" [Undecided,New]02:09
*** inc0 has quit IRC02:09
*** daneyon has joined #openstack-kolla02:10
*** dwalsh has quit IRC02:10
*** daneyon has quit IRC02:15
*** yuanying has quit IRC02:27
*** haplo37_ has quit IRC02:39
*** haplo37_ has joined #openstack-kolla02:41
*** huikang has quit IRC02:50
*** huikang has joined #openstack-kolla02:50
*** Pavo has quit IRC02:51
*** hieulq has joined #openstack-kolla02:53
*** huikang has quit IRC02:55
*** Pavo has joined #openstack-kolla02:55
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs  https://review.openstack.org/38104102:55
*** zhubingbing has joined #openstack-kolla02:56
*** jtriley has joined #openstack-kolla02:58
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs  https://review.openstack.org/38104103:03
*** hieulq has quit IRC03:04
*** huikang has joined #openstack-kolla03:08
*** huikang has quit IRC03:12
*** sdake has joined #openstack-kolla03:14
sdakeyo peeps03:14
*** hieulq has joined #openstack-kolla03:18
duonghqhi sdake03:27
sdakesup dotplus03:28
sdakeduonghq03:28
duonghqsdake: some more log about last night (in my tz) bug: https://bugs.launchpad.net/kolla/+bug/163067803:28
openstackLaunchpad bug 1630678 in kolla "keepalived container cannot get MASTER state" [Undecided,New]03:28
duonghqI'm create totally new VM for recheck03:29
sdakeoh vms03:31
sdakeyo udidn't mention vms03:31
sdakewhat is yoru virtual machine manager?03:31
duonghqiirc, I mentioned VBox03:31
duonghqmaybe wrong recall after hit my keyboard03:32
sdakei coul dnever get outgoing networkign to work in virtualbox03:37
sdakeuntil i setup a host network03:38
sdakeand did a bunch of other magic03:38
sdakewhich i now don't recall03:38
sdakei had intended to document it03:38
sdakebut never did03:38
sdakesounds environmental to me esepcially if your in a vm03:38
*** hieulq has quit IRC03:38
sdakeif that hapepned on bare metal, i'd be concerned03:38
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs  https://review.openstack.org/38104103:39
*** sdake_ has joined #openstack-kolla03:45
*** sdake has quit IRC03:47
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs  https://review.openstack.org/38104104:01
*** jmccarthy has quit IRC04:05
*** jmccarthy has joined #openstack-kolla04:06
*** haplo37 has quit IRC04:07
*** g3ek has quit IRC04:11
*** yuanying has joined #openstack-kolla04:13
*** g3ek has joined #openstack-kolla04:13
*** haplo37 has joined #openstack-kolla04:17
MarMatping sdake_04:19
sdake_sup MarMat04:19
*** eaguilar has quit IRC04:19
MarMatsdake the good thing is that cinder + ceph + magnum is working fine for me04:20
MarMatsdake_ not so good things is that there are 2 bug reports opent for magnum deployment by other people04:21
sdake_does our deploy work?04:22
sdake_we have to trust that if magnum is doa it wont be when we release04:22
*** DanyC has joined #openstack-kolla04:22
MarMatsdake_ 1st one is that reconfigure/genconfin is not working, fix for this is unfortunately not straightforward and I need some help from Jeffrey4l_04:23
MarMatsdale_ 2nd one is failure on deployment, I requested further info from reporter https://bugs.launchpad.net/kolla/+bug/163077004:23
openstackLaunchpad bug 1630770 in kolla "magnum deploy failing" [Undecided,Incomplete] - Assigned to Martin Matyáš (martinx-maty)04:23
MarMatsdake_ ^^^04:24
sdake_MarMat that secondbug looks like they are not running as sudo04:24
zhubingbinghello guys04:25
sdake_Could not create retry file '/usr/share/kolla/ansible/site.retry'. The error was: [Errno 13] Permission denied: '/usr/share/kolla/ansible/site.retry'04:25
sdake_marmat /usr/share/kolla is supposed to be read only04:25
sdake_i don't know why a site.retry file is being put in /usr/share/kolla/ansible but that is wrong04:26
MarMatsdake_ well, that's probably only side issue, main concern is "User b43ed8d94a1448e1ad876b2188d636c0 is not valid\" error04:26
MarMatsdake_ during inserting user rules04:26
sdake_marmat i see04:27
sdake_marmat we have about 6-7 days depending on how yo ucount to sort em out04:27
MarMatsdake_ but I've never seen that on my systems04:27
duonghqsdake_: for the bug, it used to work before, I will try other hypervisor04:28
duonghqsdake_: also setup host network :)04:28
MarMatsdake_ let's see, I now need cooperation of others, Jeffrey4l_ and Daviey (reporter)04:29
sdake_MarMat do you have link to the first bug04:29
MarMathttps://bugs.launchpad.net/kolla/+bug/163024804:29
openstackLaunchpad bug 1630248 in kolla "magnum genconfig fails" [High,Triaged]04:29
*** britthouser has joined #openstack-kolla04:29
*** britthouser2 has quit IRC04:30
sdake_marmat arey ou confirming https://bugs.launchpad.net/kolla/+bug/1630248 is an issue then?04:30
openstackLaunchpad bug 1630248 in kolla "magnum genconfig fails" [High,Triaged]04:30
MarMatsdake_ yes, this is confirmed issue04:31
sdake_marmat why not join the kolla-drivers team to help with bug triage04:31
*** unicell has joined #openstack-kolla04:32
MarMatsdake_ no idea what this means :-)04:32
*** unicell has quit IRC04:33
sdake_marmat the kolla-drivers team can change bug states04:33
sdake_like set them to confirmed04:33
sdake_etc04:33
MarMatsdake this I can do anyway right? I just cannot set Importance.04:34
MarMatsdake_ I already changed status of one of them04:35
*** unicell has joined #openstack-kolla04:36
sdake_i see04:36
sdake_i dont know what non-drivers can or can't do :)04:36
*** salv-orlando has joined #openstack-kolla04:41
*** yuanying has quit IRC04:45
*** salv-orlando has quit IRC04:46
*** zhubingbing has quit IRC04:49
*** Pavo has quit IRC04:51
*** unicell has quit IRC04:51
*** unicell1 has joined #openstack-kolla04:51
*** yuanying has joined #openstack-kolla04:54
*** Pavo has joined #openstack-kolla04:55
*** bjolo_ has joined #openstack-kolla05:06
*** salv-orlando has joined #openstack-kolla05:07
*** haplo37_ has quit IRC05:15
*** haplo37_ has joined #openstack-kolla05:17
*** david-lyle has joined #openstack-kolla05:19
bjolomorning05:21
*** MarMat has quit IRC05:26
*** DanyC_ has joined #openstack-kolla05:28
*** DanyC_ has quit IRC05:29
*** DanyC has quit IRC05:29
*** jtriley has quit IRC05:31
*** ldeptula has joined #openstack-kolla05:38
*** daneyon has joined #openstack-kolla05:46
*** daneyon has quit IRC05:51
*** egonzalez90 has joined #openstack-kolla06:02
*** yuanying has quit IRC06:02
*** david-lyle_ has joined #openstack-kolla06:04
*** david-lyle has quit IRC06:04
sdake_bedtime06:05
sdake_night folks06:05
sdake_hey bjolo06:05
bjolohej06:06
bjolosup06:06
sdake_tired06:06
sdake_3% batteries06:06
sdake_time for bed :)06:06
bjoloi hear ya, you are really pushing yourself06:08
bjolotry not to burn yourself out completely06:08
*** sdake_ has quit IRC06:11
*** david-lyle_ has quit IRC06:12
*** HyperJohnGraham has quit IRC06:18
*** salv-orlando has quit IRC06:29
*** dupengfei has quit IRC06:35
*** salv-orlando has joined #openstack-kolla06:36
*** tonanhngo has quit IRC06:40
*** salv-orlando has quit IRC06:41
*** athomas has joined #openstack-kolla06:44
*** g3ek has quit IRC06:46
*** tonanhngo has joined #openstack-kolla06:48
*** tonanhngo has quit IRC06:49
*** haplo37 has quit IRC06:51
*** Pavo has quit IRC06:51
*** haplo37 has joined #openstack-kolla06:51
*** g3ek has joined #openstack-kolla06:55
*** Pavo has joined #openstack-kolla06:56
*** unicell has joined #openstack-kolla06:57
*** unicell1 has quit IRC06:58
*** tonanhngo has joined #openstack-kolla07:02
*** tonanhngo has quit IRC07:03
*** b_bezak has joined #openstack-kolla07:10
*** salv-orlando has joined #openstack-kolla07:14
*** yuanying has joined #openstack-kolla07:14
*** jidar has quit IRC07:16
*** jidar_ has joined #openstack-kolla07:16
*** jidar_ is now known as jidar07:17
*** tonanhngo has joined #openstack-kolla07:17
*** tonanhngo has quit IRC07:18
*** Serlex has joined #openstack-kolla07:28
*** bachp has joined #openstack-kolla07:31
*** tonanhngo has joined #openstack-kolla07:34
*** shardy_afk is now known as shardy07:34
*** tonanhngo has quit IRC07:35
*** DanyC has joined #openstack-kolla07:36
*** DanyC has quit IRC07:40
*** bjolo_ has quit IRC07:42
*** dwalsh has joined #openstack-kolla07:44
*** unicell has quit IRC07:54
*** salv-orl_ has joined #openstack-kolla07:56
*** unicell has joined #openstack-kolla07:56
*** salv-orlando has quit IRC07:58
*** haplo37_ has quit IRC08:01
*** tovin07 has quit IRC08:03
*** tovin07 has joined #openstack-kolla08:03
*** haplo37_ has joined #openstack-kolla08:03
*** zhubingbing has joined #openstack-kolla08:09
*** zhurong has joined #openstack-kolla08:11
egonzalez90zhubingbing, ping08:12
zhubingbinghi08:12
zhubingbinghere08:12
*** mgoddard has joined #openstack-kolla08:19
*** DanyC has joined #openstack-kolla08:23
*** daneyon has joined #openstack-kolla08:29
*** mewald has joined #openstack-kolla08:31
*** daneyon has quit IRC08:34
*** Pavo has quit IRC08:51
*** Pavo has joined #openstack-kolla08:56
*** bjolo_ has joined #openstack-kolla09:02
*** tonanhngo has joined #openstack-kolla09:04
*** tonanhngo has quit IRC09:07
*** zhubingbing has quit IRC09:09
*** zhubingbing has joined #openstack-kolla09:09
wznoinskJeffrey4l_: ping09:10
duonghqzhubingbing: ping09:18
zhubingbinghere09:18
duonghqlong time no see you09:18
zhubingbing5days09:18
zhubingbing-)09:19
duonghqya09:19
*** ankush has quit IRC09:19
duonghqzhubingbing: can you give some ref?09:19
zhubingbingBack to hometown without network09:19
duonghqin your bug report and commit09:19
zhubingbing-)09:19
zhubingbingok09:20
duonghqin your bug, it's permission related but the commit is owner change09:20
zhubingbinghmm09:21
zhubingbingyes09:21
zhubingbinggnocchi user change to root09:21
*** tovin07 has quit IRC09:21
duonghqhmm, so the desc in bug report is no longer valid?09:21
*** tovin07 has joined #openstack-kolla09:22
zhubingbingyes09:22
duonghqplease update this, too09:23
zhubingbingok09:24
zhubingbinghave done09:24
*** strigazi_AFK is now known as strigazi09:25
*** tonanhngo has joined #openstack-kolla09:25
*** tonanhngo has quit IRC09:26
duonghqzhubingbing: https://bugs.launchpad.net/kolla/+bug/1586589 -> why do you mark it as invalid?09:27
openstackLaunchpad bug 1586589 in kolla "Check 127.0.1.1 in /etc/hosts on Ubuntu" [Low,Invalid] - Assigned to zhubingbing (zhubingbing)09:27
zhubingbingi have done it09:29
zhubingbingwait 5min09:29
duonghqzhubingbing: even you finished it, you should not change the pri and state09:31
duonghq*status09:31
duonghqGerrit'll changed it to fix released automatically09:32
duonghqthe importance should not be changed too, it's bug nature, doesn't depend whether the fix is released or not09:32
*** berendt has joined #openstack-kolla09:36
*** mewald1 has joined #openstack-kolla09:36
*** mewald has quit IRC09:39
*** ankush has joined #openstack-kolla09:41
zhubingbingduonghq thank you09:41
zhubingbingi know09:41
duonghqwznoinsk: aroud?09:44
wznoinskduonghq: hi09:44
duonghqfor the bug: https://bugs.launchpad.net/kolla/+bug/161329409:45
openstackLaunchpad bug 1613294 in kolla "sudo ironic-rootwrap /etc/ironic/rootwrap.conf iscsiadm -m discovery -t st -p <IP> hangs forever" [High,In progress] - Assigned to Waldemar Znoinski (wznoinsk)09:45
duonghqas I understand, it's already addressed in https://review.openstack.org/#/c/379106/409:45
zhubingbinghi09:46
zhubingbinghttps://bugs.launchpad.net/kolla/+bug/158658909:46
openstackLaunchpad bug 1586589 in kolla "Check 127.0.1.1 in /etc/hosts on Ubuntu" [High,In progress] - Assigned to zhubingbing (zhubingbing)09:46
zhubingbingI think this bug has been solved.09:46
zhubingbinghttps://github.com/openstack/kolla/blob/master/ansible/roles/prechecks/tasks/port_checks.yml#L44209:47
zhubingbingduonhq09:47
duonghqzhubingbing: let me check when the line is commit09:47
duonghqmoment09:47
zhubingbingok09:47
zhubingbingI think this bug has been solved.09:48
duonghqzhubingbing: so I think the 127.0.1.1 check should be removed? but worth another bug09:49
duonghqbut I'm not really sure the port check is neccesarry or not09:50
zhubingbingi thinks removed09:50
duonghqdid you test deploy with the check is removed?09:51
*** salv-orl_ has quit IRC09:51
duonghqmake sure the rabbitmq is up and run smoothly09:51
zhubingbingI have been tested, hostname rabbitmq parsing problem09:51
duonghqmulti-node tested?09:51
zhubingbingall-in-one09:51
duonghq1 node?09:52
zhubingbingyeah09:52
zhubingbingmulti-node no problmes09:52
zhubingbingi have tested it 3days ago09:52
duonghqI think is not sufficient, although it's should be ok as the logic is correct09:52
duonghqbut, please push a ps09:53
duonghqcan you do this, zhubingbing?09:53
duonghqremember tag the bug09:53
zhubingbingWhat I need to do?09:53
duonghqCreate a ps, which remove the 127.0.1.1 check09:54
*** egonzalez90 has quit IRC09:54
duonghqtag with this bug ID09:54
duonghqand put a ref to paul's ps in the commit msg09:54
duonghqIf it get its way to master, so the bug is automatically closed.09:54
zhubingbingok09:54
*** egonzalez90 has joined #openstack-kolla09:54
duonghqI have some trouble with the VBox VM, it's cannot deploy kolla09:55
duonghqso I cannot verify the solution quick09:55
duonghqsorry09:55
*** awiddersheim has quit IRC09:55
duonghqzhubingbing: you test on physical machine or VM?09:55
duonghq(just for information)09:56
*** msimonin has joined #openstack-kolla09:56
zhubingbingvm09:56
duonghqVBox or kvm?09:56
zhubingbingkvm09:56
duonghqfyi, the Paul commit id is 6c238ca3,09:56
duonghqI should put correspond change id to your commit for reference09:57
zhubingbingHow did you find it09:57
duonghqgit blame do the magic09:57
zhubingbing-)wow09:57
zhubingbinghttps://bugs.launchpad.net/kolla/+bug/158658909:58
openstackLaunchpad bug 1586589 in kolla "Check 127.0.1.1 in /etc/hosts on Ubuntu" [High,In progress] - Assigned to zhubingbing (zhubingbing)09:58
zhubingbingThis bug you to close it?09:58
duonghqneed a merged ps for closing this bug.09:59
duonghqyeah, this bug09:59
duonghqabsolutely09:59
zhubingbinghttps://github.com/openstack/kolla/commit/6c238ca3bfc082813252098735e3fd6062ad952e10:01
zhubingbingcommit ps10:01
duonghqwznoinsk: does the fix for https://bugs.launchpad.net/kolla/+bug/1613294 is backported to Mitaka?10:01
openstackLaunchpad bug 1613294 in kolla mitaka "sudo ironic-rootwrap /etc/ironic/rootwrap.conf iscsiadm -m discovery -t st -p <IP> hangs forever" [High,Confirmed] - Assigned to Waldemar Znoinski (wznoinsk)10:01
duonghqthanks zhubingbing10:02
zhubingbingnp10:02
zhubingbingyou done it10:02
zhubingbingclose it10:02
duonghqzhubingbing: the bug is resolved only when the check is remove from the playbook10:03
duonghqor I'll make the ps?10:03
zhubingbingI think it ok10:05
duonghqok, so I'll push the fix10:05
duonghq5min, get task jam,10:06
zhubingbingthank you10:06
zhubingbingI want to integrate cloudkitty and gnocchi10:06
zhubingbinghave you use cloud kitty?10:07
duonghqnot ready, iirc there is cloudkitty role?10:07
*** Serlex has quit IRC10:11
*** awiddersheim has joined #openstack-kolla10:12
wznoinskduonghq: sorry, in a meeting atm, fix for https://bugs.launchpad.net/kolla/+bug/1613294 was the sudoers which is now in https://review.openstack.org/#/c/379106/410:12
openstackLaunchpad bug 1613294 in kolla mitaka "sudo ironic-rootwrap /etc/ironic/rootwrap.conf iscsiadm -m discovery -t st -p <IP> hangs forever" [High,Confirmed] - Assigned to Waldemar Znoinski (wznoinsk)10:12
duonghqthank wznoinsk, marked it as fix released10:13
wznoinskduonghq: check with Jeffrey4l_ about backporting10:14
duonghqwznoinsk: will do, seem that no backport in queue yet10:15
*** tovin07 has quit IRC10:18
*** tonanhngo has joined #openstack-kolla10:19
*** tonanhngo has quit IRC10:20
*** mnasiadka has joined #openstack-kolla10:22
*** salv-orlando has joined #openstack-kolla10:23
*** zhubingbing has quit IRC10:29
*** duonghq has left #openstack-kolla10:30
*** DanyC has quit IRC10:34
*** egonzalez90 has quit IRC10:37
b_bezakhi10:41
b_bezakhow is "kolla-ansible upgrade" tested? ;)10:42
b_bezakI am preparing to upgrade from 2.0.2 to 2.0.3dev (with recent massive ubuntu-cloud archive openstack components upgrades, btw)10:42
b_bezakpushed images to local registry with new tag10:43
b_bezakhowever I am little bit worried how nova_livbvirt with running vms will behave ;)10:43
b_bezakand ceph10:43
b_bezakwell, maybe ceph not so much, compared to nova_libvirt, thougj10:44
b_bezakthough10:45
*** zhubingbing has joined #openstack-kolla10:49
*** bachp has quit IRC10:50
*** Pavo has quit IRC10:51
*** Pavo has joined #openstack-kolla10:56
*** Serlex has joined #openstack-kolla10:57
*** shardy is now known as shardy_lunch11:01
*** bachp has joined #openstack-kolla11:05
openstackgerritChristian Berendt proposed openstack/kolla: Install python-mysqldb for Ubuntu in openstack-base container  https://review.openstack.org/38239811:12
*** daneyon has joined #openstack-kolla11:12
*** ankush has quit IRC11:14
*** bachp has quit IRC11:15
*** dwalsh has quit IRC11:16
*** daneyon has quit IRC11:16
*** rhallisey has joined #openstack-kolla11:16
*** qwang has quit IRC11:17
*** qwang has joined #openstack-kolla11:17
*** msimonin has quit IRC11:20
*** bachp has joined #openstack-kolla11:20
*** msimonin has joined #openstack-kolla11:29
*** ankush has joined #openstack-kolla11:30
*** shardy_lunch is now known as shardy11:31
kfox1111sbezverk_: rhallisey: http://logs.openstack.org/41/381041/58/experimental/gate-kolla-kubernetes-ansible-workflow-ceph-nv/2b9f788/console.html11:34
kfox1111first time all the way through. :)11:34
*** eaguilar has joined #openstack-kolla11:35
kfox1111vm's launched, cinder attached, fips. :)11:35
kfox1111all out of ceph.11:35
rhalliseynice!11:35
kfox1111going to add a few more checks, some polish, and make sure its stable. bit I think its really close now. :)11:36
berendtgreat news :)11:36
*** phuongnh has quit IRC11:37
sbezverk_kfox1111: super cool!!11:38
*** egonzalez90 has joined #openstack-kolla11:43
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs  https://review.openstack.org/38104111:44
openstackgerritcaoyuan proposed openstack/kolla: test please do not merge  https://review.openstack.org/38287411:49
*** eaguilar has quit IRC11:51
*** hyakuhei- has quit IRC11:52
*** hyakuhei- has joined #openstack-kolla11:52
*** hyakuhei- has quit IRC11:52
*** hyakuhei- has joined #openstack-kolla11:52
*** hyakuhei- is now known as hyakuhei11:52
openstackgerritMauricio Lima proposed openstack/kolla: [WIP] Add HNAS as Manila backend  https://review.openstack.org/38205411:54
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla: Unmount Ceph OSD disks as part of destroy  https://review.openstack.org/38002811:59
mliimamorning guys11:59
jmccarthyIf I get this error 'The ipaddr filter requires python-netaddr be installed on the ansible controller' - what is the "ansible controller" ?12:04
berendtjmccarthy the host where you run kolla-ansible12:05
jmccarthyberendt, ok not the nodes I'm trying to deploy to, or in the toolbox container or something like this, you mean the 'operator' or 'master' node ?12:05
berendtoperator node, master node, ansible controller, ...12:06
berendtit is all the same12:06
jmccarthyOk cool, never heard of 'ansible controller' before lol12:07
berendtin ansible/roles/prechecks/tasks/port_checks.yml we use this specific filter12:07
*** zhurong has quit IRC12:07
jmccarthyYep, that's the one I have an issue in my env, will dig some more12:07
berendtit should be sufficient to simply install python-netaddr12:07
*** zhurong has joined #openstack-kolla12:08
berendti think ansible controller is a bad word for the master node.. for me a controller is something active12:08
jmccarthyand 'lo and behold - it is :)12:08
jmccarthyJust wanted to be clear where I was supposed to be installing it ;)12:08
*** mgoddard_ has joined #openstack-kolla12:08
berendtlooks like we use a different word: "deployment host"12:09
pbourkewhats meant to drag that in12:10
pbourkewe dont document to install it anywhere12:10
openstackgerritChristian Berendt proposed openstack/kolla: Add python-netaddr as required dependency on the deployment host  https://review.openstack.org/38289212:11
*** mgoddard has quit IRC12:11
berendtpbourke here you go12:12
pbourkeshould we add it to requirements.txt?12:13
berendtit is a requirement of kolla-ansible12:14
berendtto we track those requirements inside requirements.txt?12:14
berendtto = do12:14
berendtjinja2 is also listed in requirements.txt.. but it is also used by kolla-build12:14
*** mgoddard_ has quit IRC12:15
*** Jeffrey4l_ has quit IRC12:20
*** schwicht has joined #openstack-kolla12:20
*** Jeffrey4l_ has joined #openstack-kolla12:21
*** mgoddard has joined #openstack-kolla12:22
*** salv-orlando has quit IRC12:24
*** lamt has quit IRC12:25
*** srwilkers has joined #openstack-kolla12:33
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs  https://review.openstack.org/38104112:34
*** fguillot has joined #openstack-kolla12:38
mnasiadkaguys, anybody deployed a fresh master on ubuntu xenial with python 3.5?12:40
mnasiadkahttp://paste.openstack.org/show/584665/12:40
mnasiadkakolla-genpwd fails...12:40
openstackgerritzhubingbing proposed openstack/kolla: integrate gnocchi with cloudkitty Closes-Bug: #1630947  https://review.openstack.org/38291612:43
openstackbug 1630947 in kolla "cloudkitty do not support gnocchi collector" [High,In progress] https://launchpad.net/bugs/1630947 - Assigned to zhubingbing (zhubingbing)12:43
*** rodrigo_pereira has joined #openstack-kolla12:44
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs  https://review.openstack.org/38104112:47
*** srwilkers has quit IRC12:47
wznoinskJeffrey4l_: I see your change https://review.openstack.org/#/c/379106/4 assuming iscsid running somewhere, have you tested it with pxe_ipmitool maybe?12:49
*** Pavo has quit IRC12:51
*** srwilkers has joined #openstack-kolla12:51
*** Pavo has joined #openstack-kolla12:55
openstackgerritMerged openstack/kolla-kubernetes: literal block in quickstart are messed up so, blank lines added in literal block and some Inline Markup added  https://review.openstack.org/38245412:57
*** ankush has quit IRC13:01
*** jheroux has joined #openstack-kolla13:02
*** jtriley has joined #openstack-kolla13:03
*** dave-mccowan has joined #openstack-kolla13:04
*** ayoung has joined #openstack-kolla13:04
openstackgerritzhubingbing proposed openstack/kolla: integrate gnocchi with cloudkitty  https://review.openstack.org/38291613:04
*** inc0 has joined #openstack-kolla13:04
inc0good morning13:05
zhubingbingmorning13:05
*** ankush has joined #openstack-kolla13:08
inc0Jeffrey4l_, ping13:09
openstackgerritMauricio Lima proposed openstack/kolla: Remove default values from conf file  https://review.openstack.org/38293613:15
*** huikang has joined #openstack-kolla13:19
*** msimonin has quit IRC13:23
*** msimonin has joined #openstack-kolla13:24
*** mkoderer has joined #openstack-kolla13:27
*** haplo37_ has quit IRC13:28
openstackgerritzhubingbing proposed openstack/kolla: integrate gnocchi with cloudkitty  https://review.openstack.org/38291613:29
*** haplo37_ has joined #openstack-kolla13:30
*** jtriley has quit IRC13:31
*** jtriley has joined #openstack-kolla13:32
openstackgerritzhubingbing proposed openstack/kolla: integrate gnocchi with cloudkitty  https://review.openstack.org/38291613:33
sbezverk_morning, is newton out?13:35
*** huikang has quit IRC13:35
inc0sbezverk_, 12 oct13:35
mnasiadkaok, uninstalling abandoned pycrypto and installing pycryptodome did the trick13:36
*** huikang has joined #openstack-kolla13:36
sbezverk_inc0: I guess cinder folks are early to deliver it ;-)13:36
inc0nah, we're late:)13:36
sbezverk_inc0: :-) got it13:37
inc0we release later than others13:37
inc0because we depend on them:)13:37
*** jtriley has quit IRC13:37
sbezverk_inc0: I like this approach, give a bit of extra time ;-)13:38
inc0well, we need to bind to stable releases and then test ourselves out13:39
*** huikang has quit IRC13:40
*** huikang has joined #openstack-kolla13:45
*** msimonin has quit IRC13:49
*** msimonin has joined #openstack-kolla13:51
*** daneyon has joined #openstack-kolla13:54
*** ankush has quit IRC13:54
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs  https://review.openstack.org/38104113:55
*** msimonin has quit IRC13:55
*** MarMat has joined #openstack-kolla13:56
*** tonanhngo has joined #openstack-kolla13:56
Jeffrey4l_inc0, pong13:56
*** MarMat has quit IRC13:57
*** MarMat has joined #openstack-kolla13:57
inc0Jeffrey4l_, do you know if Ironic is working well?13:57
Jeffrey4l_inc0, it is. except the iscsid parted.13:58
Jeffrey4l_wznoinsk, still around? hmm, yes, iscsid should be started. and i forget to enable iscsid in the code when ironic is enabled.13:58
*** daneyon has quit IRC13:58
inc0good to know:) I'll need to deploy multi region with ironic13:59
inc0brb13:59
Jeffrey4l_inc0, cool i tested the ironic with my fix. it works fine. inc0 you should enable iscsid service too. this is not enabled automatically, now..13:59
*** sdake has joined #openstack-kolla13:59
sdakebjolo around?14:00
wznoinskJeffrey4l_: you mean like: enable_iscsid: "{{ enable_cinder_backend_iscsi | bool or enable_cinder_backend_lvm | bool }}"14:01
bjoloyup14:01
Jeffrey4l_wznoinsk, yes14:01
wznoinskJeffrey4l_: you could help me with this https://review.openstack.org/#/c/381942/14:02
bjolosdake, sup14:02
sdakebjolo re burnout last night, i am 42 years old - know how to manage myself14:03
Jeffrey4l_wznoinsk, ok. np14:03
*** pbourke has quit IRC14:03
*** pbourke has joined #openstack-kolla14:04
sdakebjolo re pushing myself -that is what I do all the time :)14:04
bjolo:)14:04
openstackgerritRyan Hallisey proposed openstack/kolla-kubernetes: Workflow Gate Testing  https://review.openstack.org/38202114:04
openstackgerritRyan Hallisey proposed openstack/kolla-kubernetes: Ansible support for Mariadb  https://review.openstack.org/33425514:04
openstackgerritRyan Hallisey proposed openstack/kolla-kubernetes: Ansible support for Rabbitmq  https://review.openstack.org/38079614:04
openstackgerritRyan Hallisey proposed openstack/kolla-kubernetes: Ansible workflow for Keystone  https://review.openstack.org/38079714:04
openstackgerritRyan Hallisey proposed openstack/kolla-kubernetes: Ansible workflow for Glance  https://review.openstack.org/38080114:04
*** ankush has joined #openstack-kolla14:04
Jeffrey4l_wznoinsk, done14:09
*** zhenguo has joined #openstack-kolla14:11
*** sdake has quit IRC14:15
*** egonzalez90 has quit IRC14:15
*** bjolo_ has quit IRC14:18
*** athomas has quit IRC14:19
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs  https://review.openstack.org/38104114:19
sbezverk_inc0: rhallisey: kfox1111: please check your mail box :-)14:23
*** neilus has joined #openstack-kolla14:23
*** athomas has joined #openstack-kolla14:25
inc0sbezverk_, 2 things14:25
inc01. when you create mariadb volume in ceph14:26
inc0try to add --shared flag14:26
inc0and see if it fixes ceph issue14:26
inc02. 11 min is horribly long, we need to figure out why it's so long14:26
inc0I'm still struggling to deploy kolla-k8s with ceph at all14:26
sbezverk_inc0: I was observing this process and it seems node failure detection was farly quick, but the process of terminating impacted pod took very long time14:27
sbezverk_node was in NotReady state but pods which were hosted by this server were in running state for at least 6-7 minutes14:28
inc0but if termination will be induced by node fail14:28
inc0like actual fail14:29
sbezverk_inc0: it supposed to be by default14:29
*** mgoddard_ has joined #openstack-kolla14:29
inc0I wonder how long that would take14:29
inc0it makes sense to try to gracefully terminate pod before migration14:29
*** mgoddard has quit IRC14:31
sbezverk_inc0: that is for sure, but when node looses power, no time for graceful termination ;-)14:31
inc0what I'm asking is if we simulate power loss, and k8s won't care for graceful termination, will it be faster14:32
inc011min is WAY too long, we need to figure out how to bring that down for demo14:32
sbezverk_inc0: that is what I did, powered off the node14:32
*** rhallisey has quit IRC14:36
*** neilus has quit IRC14:38
*** bmace has quit IRC14:39
*** bmace has joined #openstack-kolla14:39
*** jtriley has joined #openstack-kolla14:43
*** b_bezak has quit IRC14:45
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs  https://review.openstack.org/38104114:47
*** neilus has joined #openstack-kolla14:48
*** lrensing has joined #openstack-kolla14:50
*** huikang has quit IRC14:50
*** Pavo has quit IRC14:51
*** huikang has joined #openstack-kolla14:51
*** zhurong has quit IRC14:52
*** HyperJohnGraham has joined #openstack-kolla14:52
*** neilus has quit IRC14:52
*** akwasnie has quit IRC14:53
*** kong has quit IRC14:53
*** SaMnCo has quit IRC14:54
*** amaged__ has quit IRC14:55
*** DuncanT has quit IRC14:55
*** mkoderer has quit IRC14:55
*** huikang has quit IRC14:55
*** macsz has quit IRC14:55
*** mgkwill has quit IRC14:55
*** qwang has quit IRC14:55
*** peterjenkins_ has quit IRC14:55
*** ediardo has quit IRC14:55
*** smekel_ has quit IRC14:55
*** coolsvap has quit IRC14:55
*** Pavo has joined #openstack-kolla14:55
*** egonzalez90 has joined #openstack-kolla14:56
*** tmcpeak has joined #openstack-kolla14:56
*** capnoday has joined #openstack-kolla14:57
sbezverk_rhallisey: ping14:57
capnodaysup guys14:57
tmcpeaklol14:57
tmcpeako/14:58
kfox1111hi14:58
tmcpeaksecurity review time?14:58
tmcpeakinc0:14:58
*** haplo37 has quit IRC14:58
*** g3ek has quit IRC14:59
inc0yup, thanks tmcpeak14:59
*** salv-orlando has joined #openstack-kolla14:59
tmcpeakwant to hop on google hangouts?14:59
inc0Jeffrey4l_, pbourke14:59
inc0yeah, sure14:59
inc0I guess I forgot to send a link, hold on15:00
*** egonzalez90 has quit IRC15:00
inc0tmcpeak, Jeffrey4l_ pbourke https://hangouts.google.com/call/quzp3kedrbdtfcbhyts6753ihee15:00
tmcpeakcapnoday: ^15:00
*** haplo37 has joined #openstack-kolla15:01
Jeffrey4l_roger.15:01
pbourkehmm I dont have a mic handy15:01
pbourkesorry15:01
*** g3ek has joined #openstack-kolla15:01
inc0https://etherpad.openstack.org/p/kolla-threat-analysis pbourke , join if you can,even if just to listen15:03
capnoday#link https://review.openstack.org/#/c/357978/15:05
*** diogogmt has joined #openstack-kolla15:06
*** huikang has joined #openstack-kolla15:07
sbezverk_inc0: I got some ideas from kubernetes slack community, will implement it and retry my test15:07
inc0ok15:08
otavioWe have done some fixes for mitaka branch which easy the development using Vagrent. Most are backports from master. How we should submit those?15:09
*** peterjenkins_ has joined #openstack-kolla15:11
*** mnasiadka has quit IRC15:15
*** msimonin has joined #openstack-kolla15:15
*** msimonin has quit IRC15:16
*** mkoderer has joined #openstack-kolla15:21
*** dwalsh has joined #openstack-kolla15:21
*** vhosakot has joined #openstack-kolla15:21
*** akwasnie has joined #openstack-kolla15:22
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs  https://review.openstack.org/38104115:24
*** amaged__ has joined #openstack-kolla15:25
*** SaMnCo has joined #openstack-kolla15:25
*** mgkwill has joined #openstack-kolla15:27
*** kong has joined #openstack-kolla15:34
*** smekel_ has joined #openstack-kolla15:36
*** DuncanT has joined #openstack-kolla15:36
*** ccesario has quit IRC15:36
*** janem has joined #openstack-kolla15:39
janemhi all15:39
janemI get an error with deploying on 2.0.2 AIO Ubuntu host.15:40
janemstarting kolla-toolbox container15:40
janemTypeError("'NoneType' object is not iterable",)15:40
*** macsz has joined #openstack-kolla15:41
openstackgerritMauricio Lima proposed openstack/kolla: Remove unneeded blank lines  https://review.openstack.org/38304515:42
kfox1111sbezverk_: what ideas?15:42
*** ediardo has joined #openstack-kolla15:43
*** tonanhngo has quit IRC15:46
sbezverk_kfox1111: there are a couple of timers to play15:46
*** tonanhngo has joined #openstack-kolla15:46
kfox1111timers?15:46
*** coolsvap has joined #openstack-kolla15:47
*** qwang has joined #openstack-kolla15:48
*** msimonin has joined #openstack-kolla15:51
*** srwilkers has quit IRC15:53
*** diogogmt_ has joined #openstack-kolla15:53
*** diogogmt has quit IRC15:53
*** diogogmt_ is now known as diogogmt15:53
*** msimonin1 has joined #openstack-kolla15:53
*** berendt has quit IRC15:55
*** dwalsh has quit IRC15:55
*** msimonin1 has quit IRC15:55
*** msimonin has quit IRC15:55
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs  https://review.openstack.org/38104115:56
*** haplo37_ has quit IRC15:57
*** nihilifer has quit IRC15:58
*** nihilifer has joined #openstack-kolla15:58
*** haplo37_ has joined #openstack-kolla15:59
openstackgerritzhubingbing proposed openstack/kolla: fix keystone log repeated collection in heka  https://review.openstack.org/38306616:00
*** david-lyle has joined #openstack-kolla16:01
*** jheroux has quit IRC16:02
*** neilus has joined #openstack-kolla16:02
*** huikang has quit IRC16:02
*** huikang has joined #openstack-kolla16:03
*** jheroux has joined #openstack-kolla16:06
*** huikang has quit IRC16:07
*** huikang has joined #openstack-kolla16:07
*** rhallisey has joined #openstack-kolla16:07
inc0pbourke, Jeffrey4l_ can you help me fill this diagram during next couple of days16:11
inc0?16:11
pbourkeinc0: sure16:11
Jeffrey4l_inc0, np16:11
*** neilus__ has joined #openstack-kolla16:12
inc0thanks, do you understand process? whatever I did made sense?:)16:12
inc0also thanks a lot tmcpeak capnoday16:12
tmcpeakinc0: thanks!  great working with you all on this16:13
*** neilus has quit IRC16:13
Jeffrey4l_inc0, should we draw all the containers and mounts? or part of them?16:13
*** capnoday has left #openstack-kolla16:13
*** tmcpeak has left #openstack-kolla16:13
inc0Jeffrey4l_, let's start with default config16:13
Jeffrey4l_ok.16:13
inc0default services, maybe plus ceph+cinder16:13
inc0I expect this to be deployment arch that will interest most of people16:14
inc0after that we will continously work on it, but let's get defaults done first16:14
Jeffrey4l_Got it.16:14
Jeffrey4l_yes16:14
inc0in fact, after we're done we should make this required part of any patchset introducing new service:)16:15
openstackgerritzhubingbing proposed openstack/kolla: fix can't collect horizon log in heka  https://review.openstack.org/38308016:15
inc0(figure out how to make gerrit review out of it;))16:15
inc0but that's for later16:16
inc0bbiaf16:16
Jeffrey4l_gerrit for draw.io is hard i think. may be graphiv is a better tool16:16
*** strigazi is now known as strigazi_AFK16:17
*** msimonin has joined #openstack-kolla16:18
*** msimonin has quit IRC16:19
*** rstarmer has joined #openstack-kolla16:23
rstarmerhas anyone used the netapp via NFS?16:25
rstarmerfor cinder/glance16:25
*** msimonin has joined #openstack-kolla16:25
*** HyperJohnGraham has quit IRC16:27
*** zhubingbing has quit IRC16:27
*** huikang has quit IRC16:29
*** huikang has joined #openstack-kolla16:30
sbezverk_kfox1111: there are three timers I changed in kube-controller-manager which reduced running phase of a pod on a notready node considerably, not so good news is Terminating phase did not change. It seems for non petset containers it is ok as deployment or rc starts fresh copy and does not care about terminating old instance16:30
*** zhubingbing has joined #openstack-kolla16:31
pbourkeinc0: Jeffrey4l_: you guys test multinode right16:31
sbezverk_but for pet-set new instance does not get created until old is completely gone. and in our case these are most important services for us mariadb/rabbitmq and nova api16:31
*** papacz has quit IRC16:31
inc0pbourke, yeah16:31
sbezverk_kfox1111: but the worst part is second failover crashed my server :-(16:31
pbourkeinc0: have you ever seen the setup module just randomly error out in the middle of a playbook run16:31
pbourkeinc0: just dumps all facts and basically says "module failure"16:32
inc0nope..16:32
pbourkeinc0: I see it constantly to the point where it's getting hard to test anything :(16:32
zhubingbingwhat‘mater for ci gate?16:32
zhubingbinghttp://logs.openstack.org/66/383066/1/check/gate-kolla-dsvm-build-centos-binary-centos-7-nv/1153426/console.html#_2016-10-06_16_06_24_01723316:32
pbourkemust be something with my network but can't nail it down16:32
inc0pbourke, mind pasting it somewhere?16:32
Jeffrey4l_pbourke, no. need the inventory file and ansible logs16:33
*** zhenguo has quit IRC16:33
pbourkehttp://paste.openstack.org/show/584716/16:33
pbourkeits happens randomly in various plays when setup runs16:33
pbourketried loads of tweaks of ansible ssh settings16:33
inc0pbourke, never seen anything like this...16:34
*** huikang has quit IRC16:34
*** Serlex has quit IRC16:34
pbourkeif I run with -vvv it has something about "debug3: mux_client_read_packet: read header failed: Broken pipe"16:35
pbourkelike ssh is just hanging up early or something16:36
inc0pbourke, yeah, seems to be ssh issue16:36
inc0or networking16:36
*** daneyon has joined #openstack-kolla16:37
*** rstarmer has quit IRC16:38
inc0try to tweak up ssh http://askubuntu.com/questions/127369/how-to-prevent-write-failed-broken-pipe-on-ssh-connection16:38
inc0I've seen this happening during LM16:38
openstackgerritWaldemar Znoinski (wznoinsk) proposed openstack/kolla: WIP: setting basic structures for iscsi container  https://review.openstack.org/38194216:38
*** msimonin has quit IRC16:41
wznoinskJeffrey4l_: https://bugs.launchpad.net/kolla/+bug/163107216:41
openstackLaunchpad bug 1631072 in kolla "iscsid container: mkdir /sys/kernel/config: operation not permitted" [Undecided,New]16:41
zhubingbingpbourke16:42
*** daneyon has quit IRC16:42
Jeffrey4l_wznoinsk, never saw that. will check it tomorrow. need go to bed ;016:48
Jeffrey4l_;)16:48
*** msimonin has joined #openstack-kolla16:48
*** zhubingbing has quit IRC16:48
wznoinskJeffrey4l_: cheers, talk to you16:48
*** schwicht has quit IRC16:48
*** Pavo has quit IRC16:51
*** neilus__ has quit IRC16:51
*** schwicht has joined #openstack-kolla16:51
*** neilus has joined #openstack-kolla16:51
sbezverk_kfox1111: ping me when you have a chance16:55
*** Pavo has joined #openstack-kolla16:55
kfox1111whats up?16:57
sbezverk_kfox1111: have  you read  a couple of messages I sent to you above?16:57
kfox1111oh. didn't see them. sec16:58
kfox1111oh you are trying to reduce the time a terminating pod takes?16:58
kfox1111I think thats because kolla pods don't accept terms nicely and shutdown.16:59
sbezverk_kfox1111: right and I want to propose to move back to deployment for mariadb and rabbitmq16:59
kfox1111kubernetes does a 30 second  timeout by default unless yo uoverride it in the pod, or fix it to accept a term or preStop hook.16:59
sbezverk_I noticed that for RC and deployment type of pods they stuck in terminating too, but in parallel new fresh instance is running16:59
kfox1111-116:59
inc0guys how to debug "Failed to fit in any node" errors?17:00
kfox1111petsets were designed for stateful services like mariadb or rabbits that have persistance needs.17:00
sbezverk_inc0 check labeling17:00
inc0ah, yeah, makes sense17:00
kfox1111once we get to multinode mariadb/rabbit, we will really benifit from petset's pvc templating.17:01
sbezverk_kfox1111: I undersatnd but that damn 3-4 minutes terminating phase is a killer17:01
*** sdake has joined #openstack-kolla17:01
kfox1111sbezverk_: its because its being terminated imporperly.17:01
*** vhosakot_ has joined #openstack-kolla17:02
kfox1111sbezverk_: we need to add a signal handler or a preStop hook that shuts it down properly.17:02
kfox1111if you don't want to wait, just add a --now to your delete.17:02
kfox1111totally unsafe, but emediate.17:02
sbezverk_kfox1111: what difference does it make if node gets powered off???17:02
sbezverk_you can add as many hooks you want but that piece of code already not running17:03
*** david-lyle has quit IRC17:03
kfox1111not following...17:03
*** vhosakot has quit IRC17:03
kfox1111maybe we are talking about different thigns.17:03
kfox1111I'm talking about pod termination.17:03
kfox1111are you talking about nodes disapearing?17:03
sbezverk_kfox1111: here is scenario we want to show, compute node with mariadb, it gets powered off17:03
sbezverk_and mariadb moves to another compute node17:03
*** neilus has quit IRC17:04
sbezverk_it happens but very long17:04
kfox1111ah. your tryint to deal with the ceph lock thing.17:04
*** MarMat has quit IRC17:04
sbezverk_no, locking I clear right after I power down the compute node17:04
sbezverk_I mean clear manually17:04
kfox1111ok. and thats not enough to get it to reschedule?17:05
sbezverk_kfox1111: let me do step by step17:05
kfox1111or its taking a while to detect the node disapeared and fail over?17:05
sbezverk_1. power off compute node with mariadb17:05
kfox1111oik. please.17:05
sbezverk_2. clear ceph locks manually from ceph cluster17:05
sbezverk_3. wait for node not ready state takes 20-30 seconds17:06
sbezverk_4. wait for pod starts terminating with new timers takes less time17:06
*** srwilkers has joined #openstack-kolla17:06
sdakemorning again17:06
*** unicell has quit IRC17:07
sdakeboy i dislike dentistry17:07
sbezverk_5. wait for pod gets actualli terminated takes lots of time17:07
sdakei mean i like the concept17:07
sdakejust not when applied to me17:07
sbezverk_6. wait for pod to get started on new node very fast as soon as terminating phase is gone17:07
kfox1111sbezverk_: ok. and because petsets only let one exist, it has to wait until its done terminating...17:08
sbezverk_the worst part for us is 417:08
sbezverk_exactly17:08
sbezverk_rc and deployment starts new pod and that is, but not fot petsets17:08
kfox1111the ultimate solution to this is galera.17:08
kfox1111then taking a while for the one pod to come back up doesn't matter much.17:08
kfox1111and there is no downtime what so ever.17:08
sdakeinc0 about?17:10
sbezverk_kfox1111: maybe but not sure if it is doable before the summit, I would really like to explore deployment for mariadb and see if it can at least temporarely improve things17:10
inc0sdake, yeah17:10
inc0we kicked off thread analysis17:10
sdakeinc0 kfox1111  and i had a convo last night about repo split17:10
kfox1111sbezverk_: yeah, thats all fine and all. but I'm an operator that doesnt appreciate too much hand wavy hackery for demo's. I now there are other ops in the same boat.17:11
sdakeinc0 he said you are a f an of keepingkubernetes workflow in kolla-ansible repo17:11
kfox1111we shouldn't go too far out of our way to do a poor archetecture just to do a demo that looks sort of good but that ops can see through.17:11
sdakeinc0 i dont like that idea17:11
kfox1111all the demos for kubernetes so far only ever just did stuff like delete pod, whatch it come back.17:11
kfox1111but never really did it seamless.17:12
kfox1111and people eat that up.17:12
inc0sdake, why?17:12
kfox1111so we should be able to do a really good demo, with what we have, and still make a good impresion.17:12
sdakedont crater the arch to make a rockin demo17:12
sbezverk_kfox1111: well if 10 minutes is acceptabel time of ops for a service restoration on a ndoe failure17:12
inc0we have dep on kolla-ansible anyway17:12
sbezverk_then we could show it as is17:12
inc0I don't want to have ansible playbooks all over the place17:13
sdakeinc0 welcome to repo split madness17:13
inc0and kfox doesn't want to have ansible in kolla-k8s...which makes sense17:13
sdakei dont want it in kolla-ansible17:13
inc0why?17:13
sdakekolla-ansible is for orchestrating bare metal with ansible17:13
sdakenot kubernetes17:13
sdakewe have a proposal as follows17:13
kfox1111sbezverk_: we can have the script do 3 things. power off the node, kill the ceph lock, and force delete the petset. and we can tell people that its a k8s issue if asked, and they are actively working on it for 1.5 and point them at the github issue.17:14
inc0it also deploys docker with ansible17:14
sdake1. rename kolla-kubernetess to kolla-kube17:14
inc0or bifrost with ansible17:14
kfox1111sbezverk_: most ops have to recover by hand today. any auto recovery is a huge benifit.17:14
sdakeyes, it stands alone as a peice of work17:14
sdakemake a kolla-kube-ansible repo17:14
sdakemake a kolla-ansible repo17:14
sdakekeep kola named as is17:14
sdakekolla repo = containers + buidl scripts17:14
inc0ehh...you want to grow repos exponentially?17:14
sdakee.g. the core17:14
inc0first, this will be temp17:14
sdakekfox1111 how about a little help on your idea here...17:15
inc0as we aim to have k8s native thing when it becomes possible17:15
DavieyMarmat wanted me?17:15
kfox1111sdake: sure.17:15
sdakeinc0 i'd recommend reading logs toget up to speed on the thinking17:15
inc0guys, it doesn't make a remote sense imho. We have logic in kolla-ansible we could reuse17:16
rhalliseyO.o17:16
kfox1111inc0: the issue is, kolla-kubernetes is a set of templates for standing up stuff in k8s, just like kolla is a set of containers for packaging up openstack.17:16
rhalliseywe need to have the argument in person XD17:16
*** neilus has joined #openstack-kolla17:16
inc0more than happy to have it.17:16
kfox1111workflow is a seperate thing. and its quite religiously charged.17:16
inc0can I bring knife?17:16
sdakei need to know now what is planned17:16
kfox1111:)17:16
kfox1111would a hangout be better for the converastion?17:17
inc0kfox1111, workflow ideally will be handled by k8s natively...someday17:17
sdakebecaue i am presenting stuff at summit on it :)17:17
inc0before that we need something17:17
sdakeright in in the meantime we have this repo off to the side17:17
sdakewhich is the workflowbits17:17
rhalliseysdake, there will be ansible bits the will deploy17:17
inc0having separate ansible playbook in kolla-ansible makes sense - we already use kolla-ansible genconfig17:17
inc0we *need* parts of it17:17
kfox1111inc0: I think there will always be at least a little bit of workflow that will be in external systems, lilke puppet/chef, etc. enterprises will demand that. :/17:17
rhalliseysdake, it will sit in the same repo until post summit17:17
rhalliseywhere we will consider making any changes17:18
sdake rhallisey i understand that17:18
kfox1111but I agree, if k8s can handle worflw bits, it should.17:18
inc0if we separate workflow to yet another repo17:18
kfox1111but openstack's workflow is hugely complicated.17:18
sdakerhallisey however we have already made a deicsionvia vote to split the repos17:18
rhalliseysdake, yup17:18
kfox1111it may be years before k8s can handle it.17:18
rhalliseysdake, that's still needed17:18
inc0we'll need kolla-ansible to genconfig, kolla-kube-ansible to workflow and kolla-kube to well..kube17:18
inc03 projects17:18
sdakerhallisey which is still needed17:18
sdakerhallisey the vote has happened17:18
rhalliseysdake, the repo split17:19
sdakerhallisey its a done deal17:19
inc0why you don't want another file in kolla-ansible?17:19
sdakeand i am presenting it at summit17:19
rhalliseyI meant the repo split not the vote17:19
sdakeinc0 its more like 100's of files17:19
inc0no sdake17:19
kfox1111inc0: config and workflow are seperate.17:19
inc0kfox1111, only because we made it so17:19
kfox1111I may hand config my stuff, but then want to use the ansible worflow,17:19
inc0if we could access roles in kolla-ansible17:19
rhalliseysdake, we can't put kolla-k8s anisble bits into kolla-ansible without a solid plan in place17:19
inc0we can just import config.yml and it's done17:20
kfox1111or I might use genconfig to build the config, but manually deploy or use chef.17:20
sdakekolla-ansible should stand on its own17:20
rhalliseythere is no such thing at this point17:20
sdakejust as kolla stands on its own17:20
sdakeor a repo split makes no sense17:20
inc0sdake, it will stand on it's own, kolla-k8s will not stand on it's own17:20
sdakekollakube should stand on its own for same reason17:20
rhalliseythat idea is only in the ealry stages17:20
rhalliseysdake, I agree with that logic17:20
inc0if we would decouple configs to kolla-core17:20
inc0then we don't need kolla-ansible for kolla-k8s17:21
inc0and then it makes sens17:21
inc0ee17:21
sdakethen the ansible workflow related to kuberentes is going in the kube repo17:21
sdakenot the ansible repo17:21
kfox1111inc0: yeah, that woudl be idea.17:21
kfox1111ideal.17:21
inc0kfox1111, that was the plan, just didn't happen now17:21
inc0what's wrong about current solution tho? dir workflow/ansible17:21
*** neilus has quit IRC17:21
kfox1111yeah. I get that. I'm going to help with that too if I can.17:21
kfox1111inc0: perception.17:22
inc0is word ansible so dirty that even having these files on disk prevents people from using kolla?17:22
sdakeinc0 kfox1111 doesnt' want workflow in k8s17:22
kfox1111the huge stick around breaking ansible out to its own repo.17:22
sdakeand it definately doesn't belong in kolla-ansible17:22
inc0we need workflow in k8s17:22
kfox1111it would be better not to have ansible code land in kolla-kubernetes and pull it out later.17:22
kfox1111from a perception standpoint.17:22
inc0kfox1111, you don't want to end up in cross-repo dependency hell17:22
kfox1111I agree, rwe really need the workflow.17:22
inc0look how well it worked between kolla and kolla-k8s17:22
sdakeok well lets focuso on techncial roblesm not perception problems17:22
kfox1111whats there to have hell about? the ansible stuff just calls the cli.17:23
inc0agree sdake17:23
kfox1111the cli's api is very simple. :)17:23
inc0if somebody think that these yamls on disk are so dirty, I don't really care about this person17:23
inc0they can rm -rf the dir after cloning it17:23
kfox1111inc0: thats a lot of devs. :/17:23
sdakeinc0 carefu lwti that logic :)17:23
inc0issue is, it will be completely optional17:23
sdakeinc0 yamls (ansible) on disk seems to be a sore point of contention even though epoplecould just delete it17:23
sdakeinc0 no shit i argued that in the original review17:24
inc0it's like saying that I'm not going to use cinder because they have netapp driver in repo and I don't like netapp17:24
sdakeand then was told "I just destroyed my strawman argument with my argument"17:24
kfox1111inc0: yeah. thats been kolla's party line for a long time. and its bit kolla hard I think. :/17:24
sdakeit hasn't bit kollat a ll17:24
inc0it didn't17:24
sdakepeople either like ansible or htey dont17:24
sdakeif they dont - there is no option for them that is viable imo17:24
sdakebesdies RYO17:24
kfox1111people are really religious about config management tools. so much so that they stop thinking reasonably about things like, if yo udon't want to use the optional stuff, dont use it.17:25
kfox1111they spawn whole noew projects instead. :/17:25
inc0kfox1111, issue is, it will always be totally optional17:25
*** david-lyle has joined #openstack-kolla17:25
inc0if they want to be religious, let them in their zeal write puppet code for kolla-k8s17:25
kfox1111"your confusing the issue with logic" :)17:25
inc0then they can put it into dir worflow/puppet17:25
kfox1111its not a logicical issue. :)17:26
sdakewe focus on solving technical problems kfox111117:26
sdakethe technical problem (the reason we are splitting the repos) is because they are not consumable by kolla-kubernetes in an easy fashion17:26
kfox1111sdake: there is no reason to split repos on a purely technical reason.17:26
inc0kfox1111, lets not try to make world that much better for people who aren't here17:26
*** MarMat has joined #openstack-kolla17:26
inc0if somebody *really* need that and having ansible on disk is so dirty that they can't use kolla17:27
kfox1111inc0: no, I want to make it better for all of us, by not turning off devs.17:27
inc0let them come here and shout out, we can discuss then17:27
sdakewe can't be all things to all people17:27
kfox1111and ops.17:27
sdakeunless all people come up to implement all things17:27
kfox1111that makes it better for us, as we will have mor eusers, and more devs.17:27
inc0kfox1111, and 100 pararell workstreams17:27
inc0which might in effect slow our progress17:27
sdakeinc0 starting to get the picture on repo splits :)17:27
kfox1111inc0: thats not how it works. they just spawn a fuelccp or a openstack-ansible.17:27
kfox1111then we suffer by fewer devs.17:28
inc0sdake, I had this picture from day one, that's why I was opposed to it in Austin17:28
sdakekfox1111 fwiw openstack-ansible was ahead of us17:28
inc0kfox1111, they use openstack-ansible because they hate ansible?:)P17:28
sdakeinc0 i didn't get that impression17:28
inc0not that way ahead of us sdake17:28
inc0I bet we'll outrun them in release or two in terms of deployments17:28
sdakeinc0 on a totall ydifferent development path way ahead inc017:28
sdakenow we pretty much have similar features or kolla has better17:29
sdakeanyway that is irrelevant17:29
sdakejust dont say that out loud, you will ook like a dummy :)17:29
sdakefuelccp - fair to say :)17:29
inc0kfox1111, fuel-ccp will do a lot to go native k8s17:29
inc0and that's cool, we should move towards native k8s too17:29
sdakeok dudes, we all knwo native k8s is the way to go17:29
inc0when it's possible17:29
inc0before that, let's make a ansible prothesis17:30
sdakein the meantime17:30
sdakewhat inc0 said17:30
inc0put it into dir17:30
inc0in kolla-k8s or kolla-ansible17:30
sdakebut its not going into kolla-ansible ;)17:30
inc0and rm -rf when we no longer need it17:30
sbezverk_inc0: it would be great to have this prothesis well detachable and replacable ;-)17:30
inc0I'm not going to fight that battle, for me it makes sense either way17:30
kfox1111kolla-ansible is preferable to kolla-kubernetes.17:30
*** g3ek has quit IRC17:30
sdakekfox1111 - it is better to keep one working thing together17:31
sdakei understand where you want to take kollakube17:31
kfox1111I'd prefer kolla-kubernetes to be ansible free. its pretty much there already. just a few remaining bits to go.17:31
inc0putting it into kolla-ansible will remove need of genconfig and fix this perception issue17:31
sdakeit took us  3 years to get to that point in kolla itself17:31
sdakeand we still have't done the repo split17:31
inc0but will hinder the ability to develop kolla-k8s as kolla-ansible cores will have to ack any change17:31
*** unicell has joined #openstack-kolla17:31
sdakelets not try to solve that problem right up front17:31
kfox1111I agree with inc0.17:31
rhalliseyI don't want to solve it upfront either17:31
rhalliseythe project is too young17:32
kfox1111the k8s ansible workflow is really simple. I don't think the kolla core's will slow it down much.17:32
*** haplo37 has quit IRC17:32
inc0kolla-k8s path will leave some bad impressions (which I'm ok with tbh) and still genconfig from kolla-ansible until we move config templates to kolla-core17:32
sdakekfox1111 it doesn't belong in that repo17:32
sbezverk_kfox1111 +1 we do not need to deal with crazy inventory17:32
kfox1111sdake: it doens't be long in kolla-kubernetes either. :/17:32
sdakekfox1111 moreso then kolla-ansible.17:32
sdakekfox1111 your lookign at the end state17:33
sdakelets look at the stpes to get to the end state17:33
kfox1111sdake: both end state, and intermediate perception. :/17:33
inc0ok, let's just keep status quo and discuss this in Barcelona ok?17:33
sdakeperception =NOT OUR PROBLEM17:33
inc0if somebody will be rebuked by it...tough luck. we need workflow asap17:33
kfox1111inc0: do we wait on workflow till barcelona then?17:33
sdakekfox1111 workflow in kolla-k8s17:33
rhalliseykfox1111, we have session in barcelona to discuss the path forward17:33
kfox1111sdake: it is our problem, we can choose to ignore it though. :/17:33
sdakewe can sort out the history nd repo naming later17:33
inc0no, we do what Ryan, Qin and Justin are doing now17:33
*** g3ek has joined #openstack-kolla17:34
rhalliseythis is where we can figure it out17:34
inc0kfox1111, what we need to do is to encourage people to not make religious battles where they are none to be had17:34
*** haplo37 has joined #openstack-kolla17:34
kfox1111inc0: my preference is to never land it it kolla-kubernetes. by dealying, it make s that decision defacto.17:34
sdakeinc0 good luc k with that due17:34
sdake2000 years of history says you will fail :)17:34
inc0if they like puppet, let them create dir workflow/puppet and be done with it17:34
kfox1111inc0: thats impossible.17:34
inc0why kfox1111 ?17:34
kfox1111why are we splitting kolla-ansible from kolla?17:35
kfox1111why all the fighting over it if what you said is true?17:35
inc0because we do have an alternative17:35
rhalliseybeacuse there are docker containers and deployment17:35
inc0and we don't want to have 1st and 2nd class citizens17:35
rhalliseyit's in the manifesto17:35
inc0we were thinking of putting dir into kolla called "kubernetes" remember?17:36
kfox1111the same is true of workflow.17:36
kfox1111we don't want to have second class worflows.17:36
inc0it was good idea, we've chosen to not do this because ack from kolla core would hinder kolla-k8s dev17:36
kfox1111if there is one in tree, it will be considered the blessed one, and turn off others from contributing new ones.17:36
inc0that was the reason17:36
sdakekfox1111 the workflows for k8s will remain second class until they have feature parity with the kolla-ansible workflows17:36
inc0not the case for workflow17:36
inc0we're not splitting workflows until there is another one17:37
sdakeinc0 kubernetes is "another one"17:37
inc0until we have community called kolla-k8s-ansible17:37
*** tonanhngo has quit IRC17:37
inc0sdake, workflows in k8s17:37
sdakewfm17:37
sdakeworkflows in k8s = go in k8s repo17:37
inc0then this community can stand for themselves and we will discuss17:37
sdakewhatever that workflow tech may be17:37
HyperJohnGraham_Hi all17:38
sdakesup HyperJohnGraham_17:38
inc0right, k8s repo not new k8s-ansible repo17:38
kfox1111what I want to do is, ensure there is no excuse for tripleo, fuel, or other tool not to use kolla-kubernetes for the deployment piece with reguars to wrokflow and configmanagement17:38
sdakeok that wfm - i was working on answering kevin's issue as well17:38
HyperJohnGraham_sitting in meetings till noon17:38
otavioI tried to sign the CLA but the system seems out. Is there someone who could help on this?17:38
*** athomas has quit IRC17:38
inc0kfox1111, if you know anyone who would like to pick up creation of new community of puppet-workflow-lovers, let them come here;)17:38
kfox1111they may have other reasons, but the reasons should be really clear why at that point.17:38
inc0I assume you're not biased agianst ansible yourself?17:39
sdakeotavio try #openstack-infra17:39
kfox1111inc0: I may be able to get some chef devs at some point. my group is riddled with them.17:39
rhalliseykfox1111, I think we need to dev in kolla-k8s17:39
kfox1111and they are very religious about it.17:39
rhalliseythen conisder kolla-kube-anisble17:39
*** dwalsh has joined #openstack-kolla17:39
inc0kfox1111, make them start with kolla-chef17:39
sdakelets not take on any more work at this point17:39
inc0if they want to use k8s bits there - fine by me17:39
sdakewe got enough stuff to sort out17:39
kfox1111they suffere through bad deployment tools because tis what they know, and the blessed way of doing things,17:39
rhalliseythere hasn't been a release yet for it so we don't have backports or anything17:39
kfox1111rather then use better tools.17:39
inc0but let's not solve problems which might never appear17:39
rhalliseywe can stil be felexible here17:39
sdakeinc0 ya thats called tiliting at windmills :)17:40
sdakephantom threats17:40
sdakeno reason to worry about those17:40
kfox1111what is so hard about putting some ansible bits in a seperate repo?17:40
inc0kfox1111, cross-repo deps17:40
inc0release process17:40
rhalliseykfox1111, it changes the project17:40
inc0core team management17:40
rhalliseywhich we don't want to do until we have to17:40
sdakeright17:40
inc0deployment workflow (need to install 2 instead of 1)17:40
kfox1111thats pretty easy to deal with at this point I think.17:41
kfox1111cores = cores on both repos.17:41
inc0no, not easy17:41
inc0for how long?17:41
kfox1111cross repo deps, its kollakube...17:41
sdakeagree with inc0 its not easy17:41
sdakeits a permanent choice17:41
inc0also, cross-repo deps are pita17:41
kfox1111if the deps break, tis bad. we're breaking an api.17:41
kfox1111we need to ensure api anyway.17:41
kfox1111that encurages us not to break api.17:41
inc0I'd rather have this one be done by CI17:41
kfox1111release process, yeah. its an extra few steps. I'll by that.17:41
inc0not pita in gerrit17:42
*** ankush has quit IRC17:42
sdakekfox1111 inc0 is correct, it is not as easy as just making a repo17:42
inc02 different CI confs17:42
sdakelots of repos are an anti-pattern imo17:42
kfox1111I understand the pain of ci. :)17:42
sdakerepo splits happen when the current repo structure is untenable17:42
kfox1111been dealing with it a lot. :)17:42
sdakewe haven't really reached that point17:42
sdakebut we are coming to it17:42
inc0kfox1111, it's sum of all pains17:42
kfox1111if we split it, I'll setup the gate for it. how about that? :)17:43
inc0why we don't want separate repo unless there is *really* good reason for it17:43
*** tonanhngo has joined #openstack-kolla17:43
sdakesetup gate is cheap17:43
sdakegate maintenance is not cheap17:43
inc0kfox1111, no, let's get kolla-k8s working on *any* workflow before17:43
sdakekfox1111 process the above two statements with your understanding of what it takes to setup thegate ;)17:43
inc0let's get community of people advocating for split17:44
kfox1111inc0: that will probably be done today.17:44
*** tonanhngo has quit IRC17:44
inc0there is good chance that this community won't appear before k8s grow native support17:44
inc0and in fact we can have semi-native support today17:44
inc0if we write kolla-start-k8s to understand deps, much like mirantis does it in fuel-ccp17:44
inc0or SAP17:44
sdakeok so some definitions are in order17:45
rhalliseykfox1111, the workflow bits are all in a single repo at the moment17:45
kfox1111inc0: possible. also possible they just wont adopt kolla until that happens. it may be come self fulfiling.17:45
sdakekolla = containers + build (our core stuff)17:45
rhalliseythat separate repo essentially exists in that dir17:45
inc0kfox1111, I'm not writting chef code for kolla-k8s, will you?17:45
rhalliseys/repo/dir/17:45
sdakekolla-kube = everything related to kolla + kubernetes17:45
kfox1111nope. but I don't want to turn off the devs that would write a chef version.17:45
rhalliseywe will keep the workflow code seperate in the workflow dir until we deem it worth to split17:45
sdakekolla-ansible = everything related to orchestration for kolla-ansible API17:45
inc0if they will feel bad, let them come here and talk to us17:46
sdakekfox1111 phantoms imo :)17:46
kfox1111kolla-kube= the database of kolla-kube templates, ad a simple tool for managing them.17:46
sdakekfox1111 that is the end state17:46
kfox1111config/workflow is a seperate thing.17:46
sdakeit can be a seprate thing in the same repo17:46
inc0ansible is separate thing in same repo as containers17:46
kfox1111inc0: it is unknowable how many wil just turn and run rather then come and complain. but,17:46
kfox1111jusdging from my operator experience,17:47
kfox11111 person complains about an issue, and 9 just ignore it.17:47
kfox1111if you hear something from someone, you have to assume there are many more behind staying quite.17:47
inc0kfox1111, juding from my operator experience, ops are clever people, most of them won't care religiously about having a dir on disk17:47
sdakekfox1111 yup thatis a reality17:47
inc0again, case of cinder and netapp driver17:47
*** schwicht has quit IRC17:47
inc0nobody cares about having it there besides people who uses it17:48
*** ankush has joined #openstack-kolla17:48
inc0call it a workflow driver17:48
inc0change dir name to workflow_drivers or workflow_plugins17:48
inc0that should deal with perception17:48
sdakeinc0 should but doesn't17:48
sdakewe aren't here to solve perception problems, technical problems....17:48
kfox1111hmm...17:49
inc0what I'm saying, that's what we can do without hindering our ability to deliver17:49
sdakekolla-kubernetes is so green nobody will really care about perception issues17:49
inc0ability to deliver >>> perception17:49
rhalliseyagreed17:49
kfox1111seperate repo > /workflow_drivers > /workflow > /ansible17:49
sdakeya need to focus on execution rather then politicizing kolla even more then has already beendone17:49
kfox1111perception is a big deal to new prjects though. if your understaffed.17:50
inc0kfox1111, we'll deal with staff17:50
kfox1111we need to make it as easy as possible to get additional developers.17:50
sdakekfox1111 working on getting that fixed17:50
inc0don't worry17:50
*** huikang has joined #openstack-kolla17:50
inc0just give us time, it's very first summit for this project17:50
inc0when it actually exist17:50
inc0s17:50
sdakewith 3 months of dev time...17:50
kfox1111with at least 3 other projects hot on our heels. :/17:51
kfox1111its a critical time for the project.17:51
kfox1111one thing I've leared about tech...17:51
inc0yes, so let's focus on delivering it asap17:51
inc0this is how you win17:51
kfox1111its not always the best that wins.17:51
kfox1111its the first to get the general good perception.17:51
inc0not by making smoke and mirrors and making everyone happy without result at the end17:51
kfox1111and it snowballs into a clear winner.17:52
sdakeright, and kolla has a fantastic perception ingeneral17:52
*** neilus has joined #openstack-kolla17:52
sdakeall that done - IN ONE REPO17:52
rhalliseykfox1111, new devs will come17:52
inc0kfox1111, no, it will move religious war into kolla community17:52
kfox1111rhallisey: yeah, but will it be enough at the right time?17:52
inc0when chef people will be stopped by changes in shared repo that ansible people will need to review17:53
inc0look how much of a problem that was between kolla and kolla-k8s17:53
rhalliseykolla has 3 devs for the first ~8 months17:53
sdakechef not going into kolla-ansible17:53
rhalliseylook at it now17:53
sdakethe idea of deliverables is they "stand alone"17:53
sdake(in the tc sense of deliverables in projects.yaml)17:53
harlowjaqq, for folks, cause i forget, where are all the config.json files in kolla repo17:54
sdakeharlowja all over the ansible dir tree17:54
kfox1111rhallisey: I wasn't turned off by ansible. I bothered to learn a lot about it as I do use it a bit already. If I was more in the chef camp, I might have been much more turned off.17:54
inc0harlowja, https://github.com/openstack/kolla/blob/master/ansible/roles/cinder/templates/cinder-backup.json.j217:54
inc0for exampel17:54
rhalliseyharlowja, ansible/roles/<service>/defaults/config.json17:54
harlowjathx, thought i looked there, lol17:54
rhalliseynvm it's in templates17:54
*** huikang has quit IRC17:54
*** david-lyle has quit IRC17:55
sdakefind . -name config.json ;)17:55
harlowjaya ya17:55
inc0kfox1111, we all know ansible most17:55
harlowja:-P17:55
rhalliseykfox1111, I get your perspective.  But we'ere working in the *new* areas of deployment17:55
inc0we'll be turned off by chef, so we won't write code17:55
*** huikang has joined #openstack-kolla17:55
rhalliseypeople will show up to see what it is17:55
inc0if someone would like to do it, happy to accept them to community17:55
inc0let them make a dir in workflows.17:55
rhalliseywe can show multiple examples of deloyment17:55
rhalliseywhich we have curently17:55
rhalliseywe have the CLI deloyment and a workflow17:56
kfox1111inc0: I'm not asking for anyone here to write code for any other config management tool.17:56
sdakethe reason we are splitting the repos is to deliver on the "Converged datacenter"17:56
inc0kfox1111, but if we would, that would send a clear message that ansible is not only workflow17:56
sdakethats the reason kolla-k8s exists as well17:56
kfox1111just trying to make it as level a playing field technically, and perceptially for ayone wanting to do so.17:56
inc0we can make a dir for chef today17:56
inc0:)17:56
*** neilus has quit IRC17:56
rhalliseykfox1111, yes agreed.  It's up to the cores though17:56
rhalliseyand the architure we adapt17:57
rhalliseywe need to remain open and keep multiple entrpoints available17:57
inc0anyway, kfox1111 can we please agree to have this discussion on summit?17:57
kfox1111keeping the kolla-kubernetes stuff seperate from the ansible stuff, ensures that the api is at a level playing field.17:57
inc0you can bring your chef collegues to table so we can have educated discussion with another side too17:57
kfox1111inc0: I can't attend the summit.17:57
inc0ah sorry, forgot17:57
inc0invite them here then>?17:57
kfox1111and what do we do until the summit?17:57
inc0kfox1111, carry on with plan17:58
*** neilus has joined #openstack-kolla17:58
inc0I'm not making new repo before summit...I'll have to make one already17:58
rhalliseykfox1111, stick with the plan.  It will be discussed at the summit sessions we have17:58
kfox1111until the summit, does the code go in kolla-ansible, or kolla-kubernetes?17:58
sdakethat plan shouldl not include putting k8s workflow in kolla-ansible17:58
rhalliseykfox1111, kolla-k8s17:58
sdakeinc0 i can handle the repos17:58
awiddersheimspeaking of netapp17:58
sdakebut also not prior to summit17:58
inc0kfox1111, same as Ryan is doing now. kolla-k8s17:58
awiddersheimanyone have experience iwth doing netapp with kolla and NFS?17:58
kfox1111k. sound like I'm overruled. thats fine. I've said my peace. :)17:59
inc0awiddersheim, nope:(17:59
awiddersheimor NFS in general I guess17:59
awiddersheimwith kolla17:59
sdakekfox1111 i think separate repo for workflow for kolla-kube works fine17:59
rhalliseykfox1111, thank you for you opinion :)17:59
sdakekfox1111 but lets get the workflow working first ;)17:59
rhalliseykfox1111, I agree we need to disccus it17:59
sdakeand make sure kolla-k8s can stand alone17:59
*** huikang_ has joined #openstack-kolla17:59
*** huikang has quit IRC17:59
openstackgerritOtavio Salvador proposed openstack/kolla: Update Magnum dockerfiles for formatting  https://review.openstack.org/38316717:59
openstackgerritOtavio Salvador proposed openstack/kolla: Add support for magnum ubuntu binary containers  https://review.openstack.org/38316817:59
openstackgerritOtavio Salvador proposed openstack/kolla: Fix tar command path for magnum conductor using ubuntu  https://review.openstack.org/38316918:00
openstackgerritOtavio Salvador proposed openstack/kolla: Fix typos in Kolla  https://review.openstack.org/38317018:00
sdakeinc0 which new repo are you amking18:00
openstackgerritOtavio Salvador proposed openstack/kolla: Use packaged pip instead of easy_install  https://review.openstack.org/38317118:00
openstackgerritOtavio Salvador proposed openstack/kolla: Improve Vagrants bootstrap.sh proxy support  https://review.openstack.org/38317218:00
inc0I'd rather explore way to use k8s native18:00
openstackgerritOtavio Salvador proposed openstack/kolla: Refer to operator VM as primary VM  https://review.openstack.org/38317318:00
openstackgerritOtavio Salvador proposed openstack/kolla: Set internal vip to vagrant private network  https://review.openstack.org/38317418:00
openstackgerritOtavio Salvador proposed openstack/kolla: Remove unused SUPPORT_NODE variable  https://review.openstack.org/38317518:00
openstackgerritOtavio Salvador proposed openstack/kolla: Only keep localhost for 127.0.0.1  https://review.openstack.org/38317618:00
openstackgerritOtavio Salvador proposed openstack/kolla: Connect to remote hosts as root  https://review.openstack.org/38317718:00
openstackgerritOtavio Salvador proposed openstack/kolla: Use MAC address for looking for DHCP leases for Vagrant  https://review.openstack.org/38317818:00
openstackgerritOtavio Salvador proposed openstack/kolla: Fix docker daemon proxy support in vagrant  https://review.openstack.org/38317918:00
openstackgerritOtavio Salvador proposed openstack/kolla: Vagrant plugin check  https://review.openstack.org/38318018:00
kfox1111sdake: worklow working does not require code in kolla-kubernetes.18:00
rhalliseywhat18:00
kfox1111thats... unusual.18:00
kfox1111inc0: we're using native a bit. having done many an upgrade, I don't think its going to be possible for a while.18:01
kfox1111or else, we'd be implementing a workflow engine inside the containers, instead of using an existing one outside.18:01
inc0yeah upgrades with native will be hard18:02
sdake*3* months in guys18:02
sdakedont solve world hunger, solve one problem at a time :)18:02
*** neilus has quit IRC18:02
*** mewald1 has quit IRC18:02
harlowjais the whole kolla/ansible stuff still splitting off?18:02
sdakeharlowja roger18:03
sdakesame goernance18:03
sdakedifferent repos18:03
harlowjakk18:03
kfox1111yes.18:03
inc0harlowja, yeah, but we're discussing on keeping containers in kolla-core18:03
rhalliseykfox1111, let's stick with what we have for BPs for now18:03
harlowjacools18:03
inc0configs*18:03
inc0containers stays there;)18:03
sdakeand we are not renaming the kolla repo :)18:03
harlowjai'm slowly but surely making it possible to start absorbing this stuff18:03
inc0no, but I feel we need to set up nomenclature;)18:03
inc0wonderful harlowja18:04
sdakeyes - no repo renames of kolla pls18:04
sdakewe lose all fantastic analytical data in a rename18:04
sdakeand co's invest on analytics18:04
*** ppalacios has joined #openstack-kolla18:05
sdakeok so is that settled then?18:05
rhalliseyya18:05
rhalliseysummit18:05
kfox1111tabled till the summit.18:05
kfox1111workflow lands in kolla-kubernetes until then.18:06
sdakecool18:06
sdakeok now rc2 ;)18:06
*** david-lyle has joined #openstack-kolla18:06
sdakerhallisey kfox1111 we need to tag kolla-kubernetes prior to summit as well18:06
kfox1111why?18:07
rhalliseysdake, tag but no stable release right?18:07
sdakewe get kicked out of the release as an entire proejct if our deliverables don't "deliver"18:07
sdakerhallisey right just a tag18:07
rhalliseykk18:07
kfox1111ah.18:07
kfox1111k.18:07
sdakebranches happen onx releases18:07
*** ldeptula has quit IRC18:08
kfox1111I thought kolla-kubernetes was release independent though18:08
sdakekfox1111 expectations for release indepndent projects is to release on mielstones18:08
inc0kolla-k8s is18:08
inc0kolla is not18:08
sdakekollais release trailing18:08
*** HyperJohnGraham has joined #openstack-kolla18:08
kfox1111sdake: ah.18:08
sdakedid i mention i hate the dentist ;)18:08
inc0sbezverk_, kfox1111, what was command to release lock in ceph18:09
inc0?18:09
kfox1111rbd lock somethingorother.18:09
kfox1111rbd --help18:09
kfox1111its like an rbd lock ls first to get the host/token and then something like rbd unlock host token or something.18:10
inc0yeah so --shared didn't help:/18:12
kfox1111it onlys works with ro volumes18:12
*** eaguilar has joined #openstack-kolla18:13
inc0I'm half tempted to have a loop while true remove locks18:14
inc0for demo18:14
kfox1111inc0: with the ceph patch about to merge, there is only two things using rbd that will be affected by this issue.18:14
*** shardy is now known as shardy_afk18:14
kfox1111rabbit and mariadb.18:14
inc02 things we care about most18:14
kfox1111rabbit doesn't actuallly require persistance.18:14
kfox1111we could disable it.18:14
inc0we show ha18:14
inc0let's do ha18:15
kfox1111rabbit will failover to another node quickly, then the rpc's will timeout and resend.18:15
inc0if we solve it for mariadb we solve it for rabbit too18:15
kfox1111mariadb's another story.18:15
kfox1111that one must be persistant.18:15
inc0story we need to solve18:15
kfox1111we s hould just ha it.18:15
kfox1111should be doable.18:15
kfox1111brb18:15
inc0and do non-floating mariadb?:)18:15
sdakeit would be nice if you could show workflow18:16
sdakebut i know thats a stretch18:16
inc0we might just as well deploy mariadb with kolla-ansible18:16
inc0sdake, that's a goal18:16
sdakeinc0 nah we want to show a running openstack on kuberenetes18:16
sdakemariadb deployed with kolla-ansible doesn't show that18:16
inc0sdake, we are talking about different issue18:17
sdakecool18:17
inc0sbezverk_, what are things k8s folk suggested?18:17
*** HyperJohnGraham has quit IRC18:18
sdake2 weeks to sort out what kolla-kubernetes demo is18:18
kfox1111mariadb could float, a node failure though wouldn't fully autorecover yet, but it wouldn't matter.18:18
*** dwalsh has quit IRC18:18
*** schwicht has joined #openstack-kolla18:19
kfox1111the other option, if gallera doesn't like the floating ip's,18:19
kfox1111is we could just usae a daemonset for now.18:20
*** rstarmer has joined #openstack-kolla18:20
sdakekfox1111 - ntoice the message int his bug re /sys18:20
kfox1111I think we want to have that option in the toolbox anyway, as some ops will ike that more.18:20
sdakehttps://review.openstack.org/#/c/381942/218:20
sdakekfox1111 rather https://bugs.launchpad.net/bugs/163107218:21
openstackLaunchpad bug 1631072 in kolla "iscsid container: mkdir /sys/kernel/config: operation not permitted" [Undecided,New]18:21
kfox1111sdake: I found the issue on mine. it was a config option not making it into a template the way it shoudl have.18:21
sdakekfox1111 roger18:22
*** mkoderer has quit IRC18:22
kfox1111I'm guessing that one is an issue with lack of privileged flag.18:22
inc0btw I have idea for locking issue18:23
inc0.18:23
inc0we remove lock on container start18:24
inc0lock is added before container is started, it spawns, we run additional script in kolla-start to remove lock18:24
inc0dirty but works18:25
inc0can we do this?18:25
kfox1111it won't start. the volume is moutned before any user code.18:25
inc0yes, but:18:26
inc01. we do bootstrap - volume monuted, lock->118:26
inc02. we remove lock->018:26
*** tonanhngo has joined #openstack-kolla18:26
sbezverk_inc0: sorry was on the meeting18:26
inc03. we run actual bootsrap, no lock present18:26
kfox1111oh. you mean remove the lock while its still mounted?18:26
inc04.  k8s run mariadb lock->118:26
inc0yup18:27
sbezverk_inc0: they suggested to tweak a couple of timers18:27
kfox1111might work... scary as hell. :)18:27
inc0works, just tested it18:27
sbezverk_which improved things a bit18:27
wirehead_heh18:27
inc0kfox1111, yup, scary as hell, but we need to code in support for fencing in k8s18:27
inc0anyway18:27
kfox1111still. would prefer clustered mariadb to solve. no one would wnat to run the lockless way in production.18:27
kfox1111so its a jimmied up demo.18:27
sbezverk_inc0: the main issue now is behavior of petsets18:28
kfox1111I'd rather not be too fake in the demo. that's off putting.18:28
inc0kfox1111, yeah, for demo it's enough, for prod we will figure it out18:28
inc0it's not *that* fake18:28
inc0and we won't hide it18:28
rodrigo_pereiramay someone help me, I'm getting http 500 error on horizon dashboard, when running aio on vagrant environment. I'm not finding anything on logs, but i have put django on DEBUG true, I'm got this: http://pastebin.com/m8nEL4Zj18:28
sbezverk_inc0: when replacation controller or deployment controller need to start new instance instead of failed one it does not wait for old instance complete termination18:28
kfox1111we still have a huge moutn of awesome to show, without having to show off that.18:28
inc0I mean we don't presume prod-readiness yet18:28
sbezverk_which is not the case wit hpersets18:29
kfox1111sbezverk_: even more unsafe.18:29
sbezverk_oops petsets18:29
inc0kfox1111, issue with clustered db is...it might be slower than if we just deploy it old style18:29
kfox1111it shouldn't take 2 weeks to get a mariadb cluster going.18:29
*** dwalsh has joined #openstack-kolla18:29
kfox1111inc0: so? how fast does it have to be to launch a vm or 2?18:29
inc0kfox1111, what I'm saying we don't have any real value of running it on k8s without autohealinng18:30
inc0and databases weren't written with "floating around" in mind18:30
kfox1111it has value. one node won't ause the entire system to lock up waiting for it to come back.18:30
sbezverk_inc0: it we can get recovery aournd 5 minutes, would it be acceptable for now?18:30
inc0well, we need to fence it anyway18:30
kfox1111and require dangerious workarounds. :/18:31
inc0kfox1111, today18:31
inc0we need this feature in k8s, I'll write if I have to18:31
inc0everyone needs this feature, I see some code in master already18:31
kfox1111I'm saying, I think we can fix it right, for the demo. rather hten hack something together that will never fly in production.18:31
kfox1111yeah. the lock/fencing issue is in progress.18:32
kfox1111thye just build a new issue for it... let me get you a link.18:32
kfox1111https://github.com/kubernetes/kubernetes/pull/3416018:32
sbezverk_inc0: could you answer my question?18:32
kfox1111in the mean time, gallara should work.18:33
inc0sbezverk_, what takes this long? that's my question;)18:33
sbezverk_inc0: hm let me state it third time :-)18:33
*** msimonin1 has joined #openstack-kolla18:33
inc0it doesn't wait for termination18:33
inc0so it's not termination18:33
sbezverk_it takes long because petset waits for complete termination of instance before starting new one18:34
inc0even if node fails?18:34
sbezverk_yes18:34
sbezverk_node is already in notready state but isnt18:34
kfox1111thats kind of a bug too.18:34
sbezverk_instance stays in running state by default 5 minutes18:34
*** msimonin has quit IRC18:34
inc0so there is some timeout involved for it to go down?18:34
sbezverk_with new timers I managed to reduce it18:35
inc0can we make it 30s?18:35
kfox1111sbezverk_: while a bit of a hack, much more comfortable with this idea...18:35
sbezverk_but then petset container get stuck in terminating state18:35
kfox1111why not make a script that polls for dead nodes, and kills pods on them?18:35
kfox1111it can run once a second and delete the pod with --now ?18:35
inc0kfox1111, hard to kill a dead node;)18:35
sbezverk_inc0: done, but this timers do not change "terminating" part :-(18:35
kfox1111no the node, the petset pod.18:36
inc0kfox1111, if node is dead, petset pod is dead too18:36
sdakerbergeron ping18:36
inc0just k8s doesn't knwo about it18:36
inc0ehh I like this project less and less;)18:36
sbezverk_inc0: it just a figment of memory of kube brain not letting it go ;-)18:36
kfox1111inc0: its fine. your just wanting it all now. ;)18:36
sdakewelcome to distributed systems madness18:37
kfox1111it'll get there.18:37
inc0can we splatter this brain on the wall and make it go faster?;)18:37
inc0kfox1111, I know18:37
sbezverk_inc0: I filed an issue with kube this morning, so far no answer18:37
inc0just listing in my mind stuff to look at when I'd like to contribute to it18:37
kfox1111sbezverk_: what do you think? if we detect nodeDown state, will deleting the pod with --now clear it right away?18:38
*** srwilkers has quit IRC18:38
inc0that could be part of our fencing script18:38
sbezverk_kfox1111: do not think so, as when you run kuebctl delete on Terminating pod it just tells you that the pod is already deleting18:38
inc0node down -> kill all pods -> fence the crap out of ceph -> release lock -> restart18:39
kfox1111sbezverk_: --now does somethign different though.18:39
sbezverk_will try it next time I kill my compute node18:39
inc0we can make shellscript runing in look for it18:39
kfox1111sbezverk_: --now tells k8s to ignore the term/prestop hooks and hard kill the pod.18:39
inc0and say on demo "yeah, kube needs work18:40
inc0and say on demo "this is workaround, kube needs work, help needed"18:40
kfox1111I've had timeouts I've set on pods that were set to weeks, and the orchestration scripts failed to work, and killing them with --now worked.18:40
sbezverk_:-)18:40
* kfox1111 nods.18:40
sbezverk_kfox1111: are you sure they were in Terminating state?18:40
kfox1111sbezverk_: yup.18:40
kfox1111the only difference was the node was still up. so havent tried that part.18:41
inc0also ad removing lock is bad18:41
kfox1111but --now does cause it to ignore timeout on most cases. so it might on this case too.18:41
inc0it really isn't, not fencing node is bad18:41
sbezverk_kfox1111: that might change a thing ot two ;-)18:41
inc0but having lock erected for failed node - no bueno too18:41
kfox1111inc0: is there a way to fence a node in ceph?18:41
inc0yes18:41
kfox1111ok. so, how about this:18:41
kfox1111while true;18:42
inc0http://docs.ceph.com/docs/jewel/cephfs/eviction/18:42
MarMatping Jeffrey4l_18:42
kfox1111   kubectl get nodes | grep down > names18:42
kfox1111  if names |wc -l > 0 | for each node18:42
kfox1111    ceph fence node18:42
kfox1111    ceph remove all locks on node18:42
inc0ahh this is for fs18:42
kfox1111   kubectl delete pods on node --now18:42
inc0hmm, let see if it works for rbd18:42
kfox1111 sleep 118:42
kfox1111done18:42
sbezverk_kfox1111: that might solve only locking problem18:43
sbezverk_terminating mariadb container is still unsolved problem18:43
inc0kfox1111, so we can evict with cephfs18:44
*** haplo37_ has quit IRC18:45
*** haplo37_ has joined #openstack-kolla18:47
sbezverk_kfox1111: just tried and it worked with --now18:47
inc0kfox1111, can we move to cephfs driver quickly?18:49
*** harlowja has quit IRC18:50
*** Pavo has quit IRC18:51
kfox1111sbezverk_: ok, so that script above might just do the trick?18:51
kfox1111inc0: I think thats a tradeoff....18:52
kfox1111cephfs only recommends having one master.18:52
kfox1111so we tradeoff cephfs mds for mariadb.18:52
sbezverk_kfox1111: yes, for now I can even start manually to clean up all petsets containers18:52
inc0hmm18:52
sbezverk_kfox1111: never heard before what is cephfs?18:53
kfox1111sbezverk_: k. I vote for a script like that. its not very hacky and its stuff that will all go into k8s when ready.18:53
inc0so we need to figure out ceph fence node part for rbd18:53
kfox1111cephfs is a distributed filesystem built on top of the ceph object store.18:53
sbezverk_kfox1111: have nothing against script ideally would be nice to hook it somewhere ;-)18:53
kfox1111inc0: yeah. if we can solve that, the script should be easy to flesh out with the remaining details.18:54
inc0problem is it's actually going to be hard with ceph arch18:54
inc0we'd need to do iptables drop on all the osds;)18:54
inc0that has objects for this particular image18:55
inc0well..pgs..18:55
inc0for pool...you get my drift18:55
inc0but that's true for every shared storage we ever want to have if we want it to float around18:55
*** Pavo has joined #openstack-kolla18:56
*** david-lyle has quit IRC18:57
inc0one idea18:57
inc0make a rbd ro -> create cow snapshot -> mount this snapshot to new pod18:57
inc0flatten when safe18:57
*** neilus has joined #openstack-kolla19:00
inc0on the bright note sbezverk_ wirehead_ rhallisey we'll going to have stack of NUCs for talk:)19:02
inc04 of them19:02
kfox1111oh. so there isn't a ceph fence after all?19:02
inc0not for rbd19:02
inc0my bad19:02
kfox1111what about the object level? rbd just sits on top.19:02
kfox1111or were yiou thinking cephfs?19:02
*** huikang_ has quit IRC19:03
inc0cephfs has fencing - confirmed19:03
*** huikang has joined #openstack-kolla19:03
kfox1111ok... one other hack I can think of... not sure if its a good idea or not.19:03
kfox1111well... I guess that doesn't matter.19:04
kfox1111if we know the node is dead, we can just pull the lock.19:04
inc0we need to fence it19:04
inc0for prod19:04
kfox1111ok.... what if we just fence ourself?19:04
sbezverk_kfox1111: if we could somehow get notification from kubernetes event and run a script on this event that would be a good solution19:04
kfox1111take the loop above,19:04
kfox1111and change ceph fence to "ipmi power off node"19:05
inc0yeah well, that works19:05
kfox1111the script would just loop until k8s finds a dead node, and we fence it by forcing it off, then remove the lock and killing existing pods?19:05
inc0if we could make it purely on ceph level we can use k8s for it19:05
*** neilus has quit IRC19:05
rhalliseyinc0, :)19:05
kfox1111the script could run in a pod in k8s... :)19:06
kfox1111thats the k8s way. :)19:06
inc0:D19:06
inc0and it's really stateless19:06
inc0so no need for petset19:06
kfox1111and you could run 2 so one node faulure doesn't kill it.19:06
inc0we can even have couple replicas of it:)19:06
kfox1111yup.19:06
inc0I think if we could pull that off with just ceph that would be awesome, but I guess that's good science project for later:)19:07
inc0bottom line, we want stuff to float around -> we need a way to fence them19:07
*** huikang has quit IRC19:08
*** absubram has joined #openstack-kolla19:08
inc0kfox1111, one way to do ceph fencing19:09
inc0you have mariadb object19:09
inc0mariadb obj -> snap! -> new pod gets to write cow object made on top of snap! -> remove original mariadb obj19:10
*** mgoddard has joined #openstack-kolla19:10
*** mgoddard_ has quit IRC19:10
inc0old pod dies in fire as we just removed it's disk:(19:10
inc0what do you think?19:11
inc0nope, we need to stop io to it before:S19:12
*** salv-orlando has quit IRC19:13
kfox1111yeah.19:13
kfox1111I think just forcing an extra power off should let us unlock safely.19:13
inc0yeah I guess19:13
inc0ipmi poweroff rules them all19:13
kfox1111its a pretty common pattern too.19:13
kfox1111ops are use to that.19:13
inc0I know19:13
kfox1111you could even use the redhat fencing tools to do it. so it would be even more familior19:14
kfox1111the k8s folks were talking about doing the same.19:14
inc0ok, ipmi poweroff it is19:14
inc0until kube grows a fencing19:14
kfox1111this is something we probably can ship too, in production kolla-kubernetes until k8s grows it.19:15
inc0fencing pod?;)19:15
inc0yeah I guess19:15
*** mgoddard has quit IRC19:15
inc0it isn't *that* ugly19:16
inc0and will bring down db downtime on failure to seconds19:16
*** coolsvap has quit IRC19:16
sbezverk_how do you clear ceph locks from the fencing pod?19:17
sbezverk_ssh to ceph?19:17
kfox1111we have a ceph-admin pod in the ps I'm working on.19:17
inc0nah, we do container with ceph-common package and keyring19:17
kfox1111if the fencing pod has a service acount (should),19:17
kfox1111it can kubectl exec ceph-common rbd ....19:18
kfox1111it can kubectl exec ceph-admin rbd ....19:18
kfox1111ceph-admin already has an admin key19:18
inc0we can have k8s-toolbox:)19:18
inc0with ceph-admin and our fencing stuff in same container19:19
sbezverk_kfox1111 if poeple do not use ceph-admin, maybe ceph-common in this pod is good alternatives19:19
kfox1111hmm... the admin pod's not garanteed to be up though, as it could be on the fenced node.19:19
*** daneyon has joined #openstack-kolla19:19
inc0kfox1111, can we made it into multiple replicas and rc it?19:19
inc0it doesn't run anything right?19:19
sbezverk_inc0 ceph-amin is not mandatory19:19
kfox1111ok. maybe best to mount the ceph-kolla secret into the fence pod then.19:19
inc0sbezverk_, container, not package19:20
sbezverk_kfox1111 +119:20
kfox1111that keeps it nice and simple.19:20
inc0yeah19:20
sbezverk_agree mounting secret to fencing sounds very kubernetes like ;-)19:20
inc0we'd need ipmi conf and secrets in it too19:21
inc0so we need new image19:21
inc0let's publish it to kolla after we branch out neutron19:21
sbezverk_inc0: so we will need new docker in kolla19:21
sdakeone thing i'd ask19:21
kfox1111yeah. new image with ceph bits, and redhat fencing tools.19:21
sdakeis make sure anything you demo is in the queue19:21
kfox1111the rest we can pull in from configmaps/secrets.19:21
inc0sdake, sure19:22
sdakei know it can't be in the repo because the repo is frozen19:22
inc0for one more week19:22
inc0we can just not merge it till then19:22
sdakeright19:22
kfox1111+!19:22
kfox1111+119:22
inc0it's pretty elegant imho19:23
inc0at east not too horrible19:23
*** dwalsh has quit IRC19:23
kfox1111so close... got everything working in the gate, including launching vm's, setting up networking, pinging between vm's.19:23
kfox1111just gota fix cinder, and I think we're done. :)19:23
inc0yay19:23
inc0this gate will be better than kolla's;)19:24
*** daneyon has quit IRC19:24
sdakecool gate parity ;-)19:24
inc0no, kolla-ansible will actually be slightly worse;)19:24
sbezverk_kfox1111: if you are stll busy with gate, I can start fencing pod19:24
inc0we need to install ceph and cinder in kolla-ansible gate too19:24
inc0sbezverk_, let me please;)19:24
inc0or together19:24
inc0I want to have at least one patchset in kolla-k8s before summit :(19:25
kfox1111sbezverk_: yeah, please do. maybe base it on the oustanding gate ps though.19:25
sbezverk_inc0: sure as long as it does not get dealyed by other of your responsibilities ;-)19:25
kfox1111it would be nice to use the ceph-kolla secret it sets up.19:25
kfox1111as its restricted to the rbd volumes.19:25
inc0kfox1111, it sets this secret in gates tho right19:26
inc0?19:26
inc0we don't have secret_generator for it19:26
kfox1111yeah. it sets up secrets for it.19:26
inc0as we don't have ceph keyrings in passwords.yml19:26
kfox1111right.19:26
kfox1111its in the setup_gate.sh for now.19:26
kfox1111using the ceph-admin stuff it sets up.19:27
*** dwalsh has joined #openstack-kolla19:27
inc0yeah, we won't require setup gate, just kubectl create19:27
sbezverk_we need to require ceph-secret to be created in either cases19:27
kfox1111after this ps merges,19:27
inc0we need both ceph auth creds and ipmi auth creds there19:27
kfox1111I'm going to break the setup_gate into a few seperate scripts,19:27
kfox1111making it easy to run a few of the bits.19:27
sbezverk_even if ceph-admin pod is not used19:27
kfox1111like setting up the ceph-kolla secret, make the rbd voluems, etc.19:28
inc0I think I'll rather add tpls for these resources to kolla-k8s and require them in config19:28
kfox1111so we can easily call it outside the gate, until the workflow catchs up.19:28
kfox1111its a secret generation step, its gota go through a ceph client.19:28
kfox1111then the output of that is put in a template.19:28
inc0actually no, you need keys19:29
sbezverk_kfox1111: please without any dependencies :-) if I create ceph-secret manually, nothing should prevent me from using in fencing pod19:29
kfox1111yeah. that.19:29
inc0you need key to get a key:)19:29
inc0so we can just require *the key*:)19:29
kfox1111sbezverk_: you can upload ceph-secret as ceph-client-admin-key and lauch ceph-admin19:29
*** dwalsh has quit IRC19:29
kfox1111and do all the rest of the key creations / rbd volume and pool setup.19:30
sbezverk_kfox1111: my point is ceph-admin is nice to have but not mandatory and it should stay like that I think19:30
inc0yeah, I think fence pod should be self-sufficient19:30
kfox1111sbezverk_: I agree, but then you have to manually do a lot of crap in ceph to actualy support ceph.19:30
kfox1111using an admin key for it all is really bad practice.19:30
sbezverk_kfox1111: and it is my choise19:31
inc0kfox1111, but we don't need to use admin key19:31
kfox1111sure. thats fine. but don't recommend it to anyone else.19:31
inc0that doesn't mean we need to create ceph user19:31
sbezverk_cool19:31
kfox1111inc0: sure. you can manulaly create all the stuff through some other means.19:31
inc0if somebody wants to use admin- fine, but let operator just provide us with user/key19:31
kfox1111inc0: I provided the bits so you can use them if you'd like.19:31
*** dwalsh has joined #openstack-kolla19:31
inc0we require user/key for ceph and user/pass for ipmi19:32
inc0and do kubectl create based on this19:32
kfox1111and I'm going to break out the script bits so you can run that little bit of the worfklow if you'd like.19:32
kfox1111inc0: right.19:32
inc0do we have template for secret anywhere19:32
inc0?19:32
kfox1111what are you trying to do?19:33
sbezverk_inc0: the other day I sent you paste with secret template we used for ceph19:33
inc0so currently we have key for ceph in /etc/kolla-kubernetes/kolla-kubernetes.yml19:33
inc0sbezverk_, I know, but do we have it in tree?:)19:34
inc0we need 2 more secrets that won't be in original passwords.yml19:34
kfox1111inc0: usually you create secrets with kubectl secret create19:34
inc0ceph secret and ipmi secret19:34
kfox1111inc0: heres what I'm doing:19:34
kfox1111https://review.openstack.org/#/c/381041/67/tools/setup_gate.sh19:34
sbezverk_no we do not because we require operator to create them before the installation of kolla19:34
kfox1111see starting about line 389, sets up the ceph bits.19:35
*** huikang has joined #openstack-kolla19:35
inc0sbezverk_, I see...I guess we can just require them created and handle it in workflow19:35
*** janem has quit IRC19:36
inc0bbiaf, moving to home with this, I need bigger screens ;)19:36
sbezverk_inc0: yep19:36
*** eaguilar has quit IRC19:36
sbezverk_unless of course they will use Kevin's bits19:37
sbezverk_to automate it19:37
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs  https://review.openstack.org/38104119:37
kfox1111for tipical setup, we will need 4 ceph keys. one for rbd, one for nova, one for glance, one for cinder.19:39
kfox1111and if you want to use the workflow to set it up, the admin key just long enough to create the other keys.19:39
sbezverk_kfox1111: I guess that is the difference between me running cluster in the lab with one key to rule them all and a key per hobbit ;-) in real operation19:40
* kfox1111 nods.19:41
kfox1111though, a word of advice that a bunch of devs miss. if your not testing like your users actually will use it, then you run the risk of not producing useful stuff.19:42
sbezverk_sure19:42
sbezverk_make sense19:42
kfox1111shortcuts are fine for development to speed development, so long as the stuff users actually will use is still tested somehow.19:43
kfox1111and the shortcuts don't cause too much extra development fixing stuff that breaks on that side.19:43
*** inc0 has quit IRC19:45
kfox1111using stuff like admin keys or root for development can hide problems and make it really hard to debug later. you suddently see a huge pile of issues when you turn off root/adminness if you don't test all the time without it.19:45
kfox1111arg... the queue seems to be plugged.19:48
*** dave-mccowan has quit IRC19:49
sbezverk_kfox1111: I guess poeple tried to test last bits for newton before pushing it out today19:51
mliimabye guys, :)19:52
*** mliima has quit IRC19:52
*** dave-mccowan has joined #openstack-kolla19:58
*** msimonin1 has quit IRC20:00
*** msimonin has joined #openstack-kolla20:01
*** dave-mcc_ has joined #openstack-kolla20:01
*** haplo37 has quit IRC20:02
*** dave-mccowan has quit IRC20:04
*** g3ek has quit IRC20:06
*** david-lyle has joined #openstack-kolla20:06
*** inc0 has joined #openstack-kolla20:07
*** g3ek has joined #openstack-kolla20:07
*** inc0_ has joined #openstack-kolla20:07
*** inc0_ has quit IRC20:08
inc0back20:08
*** sdake has quit IRC20:09
*** dwalsh has quit IRC20:09
*** haplo37 has joined #openstack-kolla20:11
*** lrensing has quit IRC20:14
inc0oh we already have ceph secret template:)20:16
sbezverk_meeting time, bbl20:16
*** harlowja has joined #openstack-kolla20:17
inc0harlowja, when do you think you guys could start moving towards Kolla?20:17
harlowjainc0 soooooo20:17
harlowjai'm trying to get through a state of having a CI that can actually run reliably first20:17
harlowja(pre and post patch)20:18
harlowjawe don't have that; so that's my #0 step20:18
harlowja(basic things ...)20:18
inc0you mean internal CI or openstack one?20:18
harlowjainternal20:18
harlowjai'd like for that internal CI to be able to use kolla but not sure how that can happen yet20:18
harlowjaits jenkin based20:18
harlowjaand i didn't think kolla had a 'unittest' kind of task?20:19
harlowjawhere i can test before and after patching20:19
inc0well if you plan to CI build, that's gonna be tricky20:19
inc0but I'm not sure if you have to tbh20:19
harlowjaif we didn't have patches20:19
harlowjasince we do u sorta need it20:19
inc0ok, fair enough, you do know how to customize dockerfile right?20:20
harlowjayes, is there any way to tell it to apply some patches?20:20
inc0sure, hold on20:20
harlowjawithout modifying kolla docker files :-P20:20
harlowjacause i don't want more patches20:20
kfox1111harlowja: see this:20:21
inc0https://github.com/openstack/kolla/blob/master/doc/image-building.rst#dockerfile-customisation20:21
kfox1111https://review.openstack.org/#/c/381041/6820:21
*** fguillot has quit IRC20:21
kfox1111its close to merging, and is a gate job for standing it all up.20:21
inc0kfox1111, we're not talking about kolla-k8s now;)20:21
*** dwalsh has joined #openstack-kolla20:21
kfox1111oh. ok. sry.20:22
inc0harlowja, btw when you guys feel like it, k8s impl should be waiting:)20:22
harlowjakk20:24
harlowjakfox1111 perhaps someday :-P20:24
harlowjalittle steps here, ha20:24
harlowjacan only handle a little bit :)20:24
harlowjaso right now where we at is getting those post-patch unittests working btw20:24
harlowjathey weren't all passing20:24
harlowja(using venvs)20:25
harlowjaonce those work i'd be fine with doing the docker file customization20:25
*** gfidente has quit IRC20:25
inc0you can either inject some random code into certain places (we can grow more places like that as needed:))20:26
inc0modify packages, both pip and apt20:26
inc0(add, remove, total override)20:26
*** salv-orlando has joined #openstack-kolla20:27
inc0modify repo was there even in Mitaka20:27
inc0so yeah, you can do pretty much whatever you want without touching kolla code20:27
*** vhosakot_ has quit IRC20:30
*** vhosakot has joined #openstack-kolla20:31
harlowjakk20:31
harlowjabb, in meetin20:31
harlowjawill get the guy helping with CI in here soon :)20:31
harlowjacause we are nearly at post-patch stuff working (like it should be)20:31
harlowjaand then decision around 'new container' format20:31
harlowja(not rpms)20:32
*** dwalsh has quit IRC20:33
*** gfidente has joined #openstack-kolla20:39
*** gfidente has quit IRC20:39
*** gfidente has joined #openstack-kolla20:39
*** neilus has joined #openstack-kolla20:41
*** dave-mcc_ has quit IRC20:41
*** neilus has quit IRC20:46
*** MarMat has quit IRC20:47
*** Pavo has quit IRC20:51
otavioIs the master branch images working? Here it fail building the base image20:54
*** Pavo has joined #openstack-kolla20:55
*** vhosakot has quit IRC20:59
*** dave-mccowan has joined #openstack-kolla21:02
*** janem has joined #openstack-kolla21:02
*** ayoung has quit IRC21:02
*** huikang has quit IRC21:03
*** jtriley has quit IRC21:03
*** huikang has joined #openstack-kolla21:03
*** shardy_afk is now known as shardy21:05
*** huikang_ has joined #openstack-kolla21:06
*** huikang_ has quit IRC21:06
*** huikang_ has joined #openstack-kolla21:07
*** huikang has quit IRC21:08
*** huikang_ has quit IRC21:11
*** harlowja has quit IRC21:17
*** michauds has joined #openstack-kolla21:18
*** ppalacios has quit IRC21:22
*** shardy has quit IRC21:22
*** gfhellma has joined #openstack-kolla21:26
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs  https://review.openstack.org/38104121:28
openstackgerritRyan Hallisey proposed openstack/kolla-kubernetes: Workflow Gate Testing  https://review.openstack.org/38202121:30
*** schwicht has quit IRC21:31
*** david-lyle_ has joined #openstack-kolla21:34
*** david-lyle has quit IRC21:34
*** david-lyle has joined #openstack-kolla21:35
*** david-lyle has quit IRC21:35
*** david-lyle has joined #openstack-kolla21:35
*** harlowja has joined #openstack-kolla21:39
*** harlowja has quit IRC21:39
*** david-lyle_ has quit IRC21:39
*** rstarmer has quit IRC21:39
*** harlowja has joined #openstack-kolla21:39
*** tonanhngo has quit IRC21:49
*** jheroux has quit IRC21:49
*** tonanhngo has joined #openstack-kolla21:51
*** neilus has joined #openstack-kolla21:54
*** david-lyle has quit IRC21:56
*** tonanhngo has quit IRC21:56
*** tonanhngo has joined #openstack-kolla21:57
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs  https://review.openstack.org/38104121:58
*** neilus has quit IRC21:59
harlowjainc0 so after CI going somewhat ok22:00
harlowjathen ya, docker stuff becomes the next question22:00
harlowjaand then figuring out what to do with all of our puppet (that will likely fall apart right around the docker time-period)22:00
harlowjai'm hoping that perhaps kolla-ansible will be split off around then22:01
harlowjaand we can use kolla-core (or whatever its called)22:01
inc0kolla ansible is going to be split right after summit22:01
harlowjakk22:01
*** absubram has quit IRC22:01
harlowjashould align good i think then22:01
*** daneyon has joined #openstack-kolla22:02
harlowjahas anyone been running kolla (image building, basic testing) in jenkins ?22:02
*** tonanhngo has quit IRC22:02
harlowjaor tried doing that in a automated fashion22:02
* harlowja just curios22:02
harlowjado people know if https://sdake.io/2015/11/11/the-tldr-on-immutable-infrastructure/ is still valid ?22:05
harlowja'With Kolla we started with #2, briefly tried #4, and finished with technique #3.'22:05
harlowjais that still true?22:05
*** daneyon has quit IRC22:06
kfox1111harlowja: I think is configurable now.22:09
harlowjakk22:09
*** sdake has joined #openstack-kolla22:13
*** ayoung has joined #openstack-kolla22:21
openstackgerritRyan Hallisey proposed openstack/kolla-kubernetes: Workflow Gate Testing  https://review.openstack.org/38202122:22
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs  https://review.openstack.org/38104122:34
*** jtriley has joined #openstack-kolla22:37
*** rhallisey has quit IRC22:40
*** rhallisey has joined #openstack-kolla22:40
*** rhallisey has quit IRC22:40
*** rhallisey has joined #openstack-kolla22:41
*** rhallisey has quit IRC22:41
*** jtriley has quit IRC22:42
*** g3ek has quit IRC22:44
*** michauds has quit IRC22:44
*** g3ek has joined #openstack-kolla22:46
*** Pavo has quit IRC22:50
*** Pavo has joined #openstack-kolla22:51
*** MarMat has joined #openstack-kolla22:52
kfox1111"Horizon no longer requires Nova (or Glance) to function; it will run as long as keystone is present (for instance, swift-only deployments)." interesting. :)22:53
*** salv-orlando has quit IRC23:01
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs  https://review.openstack.org/38104123:05
*** ayoung has quit IRC23:05
*** portdirect has joined #openstack-kolla23:12
*** gfhellma has quit IRC23:14
wirehead_It largely worked when I had it running in a Nova-and-Glance-free environment in the very early days of kube-k8s.23:15
*** dwalsh has joined #openstack-kolla23:18
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs  https://review.openstack.org/38104123:27
kfox1111sbezverk_: http://docs.openstack.org/releasenotes/cinder/newton.html see the known issues section.23:29
kfox1111probably will affect the iscis driver.23:29
hogepodgeso, sorry for what will certainly be a silly question, but on a multi-node deployment how do I restart the rabbit service (containers)?23:30
sbezverk_kfox1111: interesting23:31
hogepodgeah, nevermind, docker restart does the trick :-P23:38
*** rodrigo_pereira has quit IRC23:40
harlowjaanother intersting question, for questions like hogepodge is it thought that all similar types of scripts should end up as ansible in say https://github.com/openstack/kolla/tree/master/ansible/roles/rabbitmq/tasks23:42
harlowjalike restart.yml or something should go there and ...23:42
harlowjaas a collection of all things operators may need23:42
hogepodgeharlowja: ah23:43
hogepodgeyes23:43
harlowjaseems sorta like it? i'm not sure the answer :-P23:43
hogepodgeas it stands I have to redeploy. rabbit is sooooo picky about hostnames23:43
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs  https://review.openstack.org/38104123:44
*** dwalsh has quit IRC23:45
*** dwalsh has joined #openstack-kolla23:49
*** schwicht has joined #openstack-kolla23:49
*** ayoung has joined #openstack-kolla23:50
*** ankush has quit IRC23:53
kfox1111heh. heat finally gained conditionals.23:57
*** ankush has joined #openstack-kolla23:58
harlowjais it turing complete yet?23:58
harlowjathats usually when u know that <u shouldn't be using this> lol23:59
*** dwalsh has quit IRC23:59
kfox1111no. I argued hard it should never be.23:59
*** rbbratta has joined #openstack-kolla23:59
kfox1111:)23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!