Saturday, 2016-10-15

*** haplo37_ has quit IRC00:00
*** haplo37_ has joined #openstack-kolla00:02
kfox1111nice.... got selenium working nicely in k8s. :)00:03
*** tonanhngo has quit IRC00:07
*** hieulq__ has quit IRC00:14
*** hieulq_ has joined #openstack-kolla00:14
*** tonanhngo has joined #openstack-kolla00:19
*** tonanhngo has quit IRC00:24
*** Pavo has quit IRC00:24
*** Pavo has joined #openstack-kolla00:26
*** dave-mccowan has joined #openstack-kolla00:26
*** dave-mcc_ has joined #openstack-kolla00:30
*** Pavo has quit IRC00:30
*** dave-mccowan has quit IRC00:32
*** Pavo has joined #openstack-kolla00:32
*** salv-orlando has joined #openstack-kolla00:33
*** salv-orlando has quit IRC00:38
*** huikang has joined #openstack-kolla00:39
*** Pavo has quit IRC00:43
*** Pavo has joined #openstack-kolla00:44
sbezverkkfox1111: man how long you are going to torture my ps. I think it is clear now that it is not the issue, the issue is with the gate..00:48
*** huikang has quit IRC00:58
*** huikang has joined #openstack-kolla00:58
*** tonanhngo has joined #openstack-kolla00:59
*** tonanhngo has quit IRC01:00
*** huikang has quit IRC01:03
*** huikang has joined #openstack-kolla01:08
*** Pavo_ has joined #openstack-kolla01:08
*** Pavo has quit IRC01:11
*** Pavo_ is now known as Pavo01:11
*** tonanhngo has joined #openstack-kolla01:19
*** tonanhngo has quit IRC01:21
*** janem has joined #openstack-kolla01:24
*** dave-mcc_ has quit IRC01:26
*** Pavo has quit IRC01:26
*** Pavo has joined #openstack-kolla01:28
*** tonanhngo has joined #openstack-kolla01:34
*** salv-orlando has joined #openstack-kolla01:35
*** tonanhngo has quit IRC01:35
*** Pavo has quit IRC01:35
*** salv-orlando has quit IRC01:39
*** Pavo has joined #openstack-kolla01:39
*** HyperJohnGraham_ has joined #openstack-kolla01:52
HyperJohnGraham_hi all01:53
HyperJohnGraham_i just tripped over this link http://www.infoworld.com/article/3130822/linux/ubuntu-1610-ups-cloud-ante-with-kubernetes-openstack.html01:53
*** dave-mccowan has joined #openstack-kolla01:59
*** MarMat_ has joined #openstack-kolla02:02
*** tonanhngo has joined #openstack-kolla02:07
*** tonanhngo has quit IRC02:08
*** janem has quit IRC02:21
*** dave-mccowan has quit IRC02:22
*** eaguilar has quit IRC02:30
*** tonanhngo has joined #openstack-kolla02:33
*** tonanhngo has quit IRC02:34
*** salv-orlando has joined #openstack-kolla02:36
*** spsingh has joined #openstack-kolla02:38
*** salv-orlando has quit IRC02:40
*** neilus has joined #openstack-kolla02:48
*** neilus has quit IRC02:53
*** tonanhngo has joined #openstack-kolla02:53
*** tonanhngo has quit IRC02:55
*** bjolo_ has joined #openstack-kolla02:56
*** haplo37_ has quit IRC02:59
bjolo_Morning03:00
*** haplo37_ has joined #openstack-kolla03:01
*** tonanhngo has joined #openstack-kolla03:07
*** tonanhngo has quit IRC03:10
*** tonanhngo has joined #openstack-kolla03:14
*** zhubingbing has joined #openstack-kolla03:17
zhubingbingping <otavio>03:18
zhubingbingping otavio03:18
zhubingbingtrove next week, i will to do it03:18
openstackgerritzhubingbing proposed openstack/kolla: add kuryr to precheck  https://review.openstack.org/37837503:29
*** salv-orlando has joined #openstack-kolla03:36
*** salv-orlando has quit IRC03:41
openstackgerritzhubingbing proposed openstack/kolla: Fix heat upgrade permission issue  https://review.openstack.org/38651203:56
*** unicell has joined #openstack-kolla04:08
openstackgerritcaoyuan proposed openstack/kolla: Delete the unnecessary space  https://review.openstack.org/38688704:09
*** MarMat_ has quit IRC04:15
*** zhubingbing has quit IRC04:18
*** zhubingbing has joined #openstack-kolla04:18
*** unicell1 has joined #openstack-kolla04:23
*** unicell has quit IRC04:23
*** salv-orlando has joined #openstack-kolla04:37
*** neilus has joined #openstack-kolla04:38
*** salv-orlando has quit IRC04:42
*** neilus has quit IRC04:43
*** ayoung has quit IRC04:47
*** g3ek has quit IRC04:49
*** Pavo has quit IRC04:49
*** haplo37 has quit IRC04:51
*** Pavo has joined #openstack-kolla04:52
*** haplo37 has joined #openstack-kolla04:54
*** g3ek has joined #openstack-kolla04:57
*** huikang has quit IRC04:58
*** Pavo has quit IRC04:59
*** huikang has joined #openstack-kolla04:59
*** Pavo has joined #openstack-kolla04:59
*** huikang has quit IRC05:03
*** Pavo has quit IRC05:08
*** zhubingbing has quit IRC05:10
*** Pavo has joined #openstack-kolla05:10
*** zhubingbing has joined #openstack-kolla05:17
*** zhubingbing has quit IRC05:22
*** zhubingbing has joined #openstack-kolla05:25
*** salv-orlando has joined #openstack-kolla05:38
*** salv-orlando has quit IRC05:43
openstackgerritzhubingbing proposed openstack/kolla: Fix congress upgrade permission issue  https://review.openstack.org/38690005:53
*** neilus has joined #openstack-kolla06:00
*** neilus has quit IRC06:04
openstackgerritprameswar proposed openstack/kolla:  apt instead of apt-get to install packages  in Ubuntu  https://review.openstack.org/37947706:07
*** zhubingbing has quit IRC06:12
*** salv-orlando has joined #openstack-kolla06:39
*** tonanhngo has quit IRC06:42
*** salv-orlando has quit IRC06:44
*** tonanhngo has joined #openstack-kolla06:53
*** tonanhngo has quit IRC06:56
openstackgerritprameswar proposed openstack/kolla:  apt instead of apt-get to install packages  in Ubuntu  https://review.openstack.org/37947707:03
*** tonanhngo has joined #openstack-kolla07:09
*** tonanhngo has quit IRC07:11
*** lamt has joined #openstack-kolla07:20
*** msimonin has joined #openstack-kolla07:21
*** spsingh has quit IRC07:21
*** msimonin has quit IRC07:23
*** tonanhngo has joined #openstack-kolla07:25
*** tonanhngo has quit IRC07:27
*** neilus has joined #openstack-kolla07:33
*** neilus has quit IRC07:34
*** tonanhngo has joined #openstack-kolla07:45
*** tonanhngo has quit IRC07:46
*** sdake has joined #openstack-kolla08:03
*** sdake has quit IRC08:05
*** neilus has joined #openstack-kolla08:18
*** qwang_ has joined #openstack-kolla08:23
*** neilus has quit IRC08:23
*** rbbratta1 has joined #openstack-kolla08:25
*** g3ek- has joined #openstack-kolla08:27
openstackgerritMathias proposed openstack/kolla: Mariadb performance tuning  https://review.openstack.org/38613008:31
*** msimonin has joined #openstack-kolla08:33
*** g3ek has quit IRC08:33
*** haplo37 has quit IRC08:33
*** rbbratta has quit IRC08:33
*** qwang has quit IRC08:33
*** goldyfruit has quit IRC08:33
*** awiddersheim has quit IRC08:33
*** quasar has quit IRC08:33
*** zeus has quit IRC08:33
*** hogepodge has quit IRC08:33
*** rbergeron has quit IRC08:33
*** kfox1111 has quit IRC08:33
*** Kdecherf has quit IRC08:33
*** akscram has quit IRC08:33
*** dcwangmit01 has quit IRC08:33
*** yee379 has quit IRC08:33
*** g3ek- is now known as g3ek08:33
*** qwang_ is now known as qwang08:33
*** msimonin has quit IRC08:33
*** awiddersheim has joined #openstack-kolla08:35
openstackgerritMathias proposed openstack/kolla: Set memcache connection limit to 5000  https://review.openstack.org/38610508:38
*** akscram has joined #openstack-kolla08:38
*** salv-orlando has joined #openstack-kolla08:41
*** salv-orlando has quit IRC08:45
*** zhubingbing has joined #openstack-kolla08:46
*** goldyfruit has joined #openstack-kolla08:48
*** haplo37 has joined #openstack-kolla08:51
*** quasar has joined #openstack-kolla08:51
*** zeus has joined #openstack-kolla08:51
*** hogepodge has joined #openstack-kolla08:51
*** rbergeron has joined #openstack-kolla08:51
*** kfox1111 has joined #openstack-kolla08:51
*** dcwangmit01 has joined #openstack-kolla08:51
*** yee379 has joined #openstack-kolla08:51
*** yee379 has quit IRC08:51
*** yee379 has joined #openstack-kolla08:51
*** neilus has joined #openstack-kolla08:56
*** spsingh has joined #openstack-kolla08:57
*** zhubingbing has quit IRC08:59
*** neilus has quit IRC09:01
*** zhubingbing has joined #openstack-kolla09:03
*** zhubingbing_ has joined #openstack-kolla09:06
*** Kdecherf has joined #openstack-kolla09:06
*** msimonin has joined #openstack-kolla09:08
*** zhubingbing has quit IRC09:09
openstackgerritzhubingbing proposed openstack/kolla: Fix mistral upgrade permission issue  https://review.openstack.org/38691709:16
openstackgerritzhubingbing proposed openstack/kolla: Fix manila upgrade permission issue  https://review.openstack.org/38692009:25
*** salv-orlando has joined #openstack-kolla09:26
*** dcwangmit01_ has joined #openstack-kolla09:27
*** quasar__ has joined #openstack-kolla09:32
*** rbergero1 has joined #openstack-kolla09:33
*** kfox1111_ has joined #openstack-kolla09:34
openstackgerritzhubingbing proposed openstack/kolla: Fix murano upgrade permission issue  https://review.openstack.org/38692209:35
*** tonanhngo has joined #openstack-kolla09:35
*** salv-orlando has quit IRC09:35
*** haplo37- has joined #openstack-kolla09:36
openstackgerritcaoyuan proposed openstack/kolla: Fix watcher upgrade permission issue  https://review.openstack.org/38692309:36
*** hogepodge_ has joined #openstack-kolla09:37
*** tonanhngo has quit IRC09:37
*** haplo37 has quit IRC09:37
*** quasar has quit IRC09:37
*** zeus has quit IRC09:37
*** hogepodge has quit IRC09:37
*** rbergeron has quit IRC09:37
*** kfox1111 has quit IRC09:37
*** dcwangmit01 has quit IRC09:37
*** msimonin has quit IRC09:37
*** haplo37- is now known as haplo3709:37
*** msimonin has joined #openstack-kolla09:37
*** msimonin has quit IRC09:38
*** zeus has joined #openstack-kolla09:39
*** msimonin has joined #openstack-kolla09:39
*** zeus is now known as Guest9682809:39
openstackgerritcaoyuan proposed openstack/kolla: Fix rally upgrade permission issue  https://review.openstack.org/38692709:44
openstackgerritzhubingbing proposed openstack/kolla: Fix sahara upgrade permission issue  https://review.openstack.org/38692809:47
openstackgerritcaoyuan proposed openstack/kolla: Fix watcher upgrade permission issue  https://review.openstack.org/38692309:48
*** msimonin has quit IRC09:50
openstackgerritzhubingbing proposed openstack/kolla: Fix cinder upgrade permission issue  https://review.openstack.org/38693009:54
*** tonanhngo has joined #openstack-kolla09:55
openstackgerritcaoyuan proposed openstack/kolla: Fix rally upgrade permission issue  https://review.openstack.org/38692709:55
*** tonanhngo has quit IRC09:55
openstackgerritcaoyuan proposed openstack/kolla: change perm 0644 to 0600 in murano role  https://review.openstack.org/38693209:59
openstackgerritzhubingbing proposed openstack/kolla: Fix magnum upgrade permission issue  https://review.openstack.org/38693410:05
*** tonanhngo has joined #openstack-kolla10:14
*** tonanhngo has quit IRC10:17
openstackgerritcaoyuan proposed openstack/kolla: Fix watcher upgrade permission issue  https://review.openstack.org/38692310:27
*** salv-orlando has joined #openstack-kolla10:32
openstackgerritcaoyuan proposed openstack/kolla: Fix watcher upgrade permission issue  https://review.openstack.org/38692310:34
*** salv-orlando has quit IRC10:37
*** dwalsh has joined #openstack-kolla10:42
*** neilus has joined #openstack-kolla10:44
openstackgerritcaoyuan proposed openstack/kolla: Fix telegraf upgrade permission issue  https://review.openstack.org/38693510:45
openstackgerritzhubingbing proposed openstack/kolla: Fix haproxy upgrade permission issue  https://review.openstack.org/38650010:46
openstackgerritzhubingbing proposed openstack/kolla: Fix heat upgrade permission issue  https://review.openstack.org/38651210:46
*** neilus has quit IRC10:48
openstackgerritzhubingbing proposed openstack/kolla: Fix heat upgrade permission issue  https://review.openstack.org/38651210:50
openstackgerritzhubingbing proposed openstack/kolla: Fix elasticsearch upgrade permission issue  https://review.openstack.org/38652310:50
openstackgerritzhubingbing proposed openstack/kolla: Fix congress upgrade permission issue  https://review.openstack.org/38690010:53
openstackgerritzhubingbing proposed openstack/kolla: Fix manila upgrade permission issue  https://review.openstack.org/38692010:58
*** portdirect has quit IRC11:01
*** portdirect has joined #openstack-kolla11:02
*** portdirect has quit IRC11:03
openstackgerritzhubingbing proposed openstack/kolla: Fix murano upgrade permission issue  https://review.openstack.org/38692211:03
*** portdirect_ has joined #openstack-kolla11:03
*** portdirect_ is now known as portdirect11:04
openstackgerritzhubingbing proposed openstack/kolla: Fix mistral upgrade permission issue  https://review.openstack.org/38691711:08
openstackgerritzhubingbing proposed openstack/kolla: Fix sahara upgrade permission issue  https://review.openstack.org/38692811:12
openstackgerritzhubingbing proposed openstack/kolla: Fix cinder upgrade permission issue  https://review.openstack.org/38693011:16
openstackgerritzhubingbing proposed openstack/kolla: Fix magnum upgrade permission issue  https://review.openstack.org/38693411:20
*** tonanhngo has joined #openstack-kolla11:23
*** msimonin has joined #openstack-kolla11:24
*** tonanhngo has quit IRC11:24
*** zhurong has joined #openstack-kolla11:25
*** msimonin has quit IRC11:29
*** salv-orlando has joined #openstack-kolla11:33
*** salv-orlando has quit IRC11:37
*** tonanhngo has joined #openstack-kolla11:45
*** portdirect has quit IRC11:45
*** tonanhngo has quit IRC11:45
*** SPSINGH__ has joined #openstack-kolla11:51
*** spsingh has quit IRC11:52
*** portdirect has joined #openstack-kolla11:58
openstackgerritzhubingbing proposed openstack/kolla: Fix heat upgrade permission issue  https://review.openstack.org/38651212:00
openstackgerritzhubingbing proposed openstack/kolla: Fix elasticsearch upgrade permission issue  https://review.openstack.org/38652312:02
*** dwalsh has quit IRC12:06
*** msimonin has joined #openstack-kolla12:10
*** tonanhngo has joined #openstack-kolla12:15
*** tonanhngo has quit IRC12:16
*** zhubingbing_ has quit IRC12:17
*** msimonin has left #openstack-kolla12:19
*** haplo37 has quit IRC12:21
*** g3ek has quit IRC12:21
*** g3ek has joined #openstack-kolla12:23
*** haplo37 has joined #openstack-kolla12:23
*** zhubingbing_ has joined #openstack-kolla12:24
*** salv-orlando has joined #openstack-kolla12:34
*** zhubingbing_ has quit IRC12:35
*** salv-orlando has quit IRC12:38
*** zhubingbing_ has joined #openstack-kolla12:40
*** tonanhngo has joined #openstack-kolla12:44
*** tonanhngo has quit IRC12:46
*** SPSINGH__ has quit IRC12:49
*** Pavo has quit IRC12:53
*** Pavo has joined #openstack-kolla12:58
*** tonanhngo has joined #openstack-kolla13:05
*** tonanhngo has quit IRC13:06
*** SPSINGH__ has joined #openstack-kolla13:06
*** sdake has joined #openstack-kolla13:13
zhubingbing_sup sdake13:17
*** sdake has quit IRC13:19
*** denaitre has quit IRC13:28
*** neilus has joined #openstack-kolla13:31
*** tonanhngo has joined #openstack-kolla13:34
*** salv-orlando has joined #openstack-kolla13:35
*** tonanhngo has quit IRC13:35
*** neilus has quit IRC13:36
*** salv-orlando has quit IRC13:39
*** haplo37_ has quit IRC13:50
*** haplo37_ has joined #openstack-kolla13:52
*** sdake has joined #openstack-kolla14:02
*** rbergero1 is now known as rbergeron14:04
*** rbergeron has joined #openstack-kolla14:04
*** sdake has quit IRC14:07
*** msimonin has joined #openstack-kolla14:09
*** msimonin has quit IRC14:09
*** sdake has joined #openstack-kolla14:17
*** tonanhngo has joined #openstack-kolla14:17
*** tonanhngo has quit IRC14:18
*** msimonin has joined #openstack-kolla14:18
*** msimonin has quit IRC14:23
*** sdake has quit IRC14:27
*** tonanhngo has joined #openstack-kolla14:35
*** salv-orlando has joined #openstack-kolla14:35
*** tonanhngo has quit IRC14:36
*** salv-orlando has quit IRC14:40
*** huikang has joined #openstack-kolla14:43
*** huikang has quit IRC14:43
*** huikang has joined #openstack-kolla14:44
*** huikang has quit IRC14:48
*** eaguilar has joined #openstack-kolla14:55
*** huikang has joined #openstack-kolla15:17
*** zhubingbing_ has quit IRC15:18
*** sdake has joined #openstack-kolla15:27
*** salv-orlando has joined #openstack-kolla15:36
kfox1111_sbezverk: alive?15:37
*** kfox1111_ is now known as kfox111115:37
*** v1k0d3n has quit IRC15:39
*** salv-orlando has quit IRC15:41
*** msimonin has joined #openstack-kolla15:45
*** msimonin has quit IRC15:47
sbezverkkfox1111_: yep15:47
kfox1111there's something different between the two systems.. we need to narrow it down.15:51
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: Fixing cinder v1 and v2 endpoints  https://review.openstack.org/38661915:54
*** zhurong has quit IRC15:54
kfox1111I'm thinking maybe kolla genconfig's telling it the wrong enpoint. looking at what it generates, its specifying cinder v2.15:56
kfox1111v115:56
kfox1111I just forced it to v2. lets see if that fixes it.15:58
sbezverkkfox1111: I am not sure getconifg touches endpoints15:58
sbezverkit is generated during the bootstrap process.15:58
kfox1111http://logs.openstack.org/77/386777/2/experimental/gate-kolla-kubernetes-ansible-workflow-ceph-nv/62256c0/logs/kolla/nova-api/nova.conf15:58
kfox1111look at the cinder section. its specifying to nova where to lookup the endpoint, and it looks like a v1 flag to me.15:59
sbezverkok, you were talking about nova..15:59
kfox1111yeah. and its nova thats' failing to talk to cinder sometimes.15:59
kfox1111so the endpoint changing from v2 to v1 is likely to be an issue.16:00
kfox1111its odd though that you never see the problem.16:00
kfox1111it may be that ceph calls back to cinder-api and iscsi doesn't in one bit of code.16:00
*** sdake has quit IRC16:01
kfox1111this may be a bug in kolla genconfig actually. if its specifying a v1 endpoint and things only work with a v2 one.16:02
kfox1111I don't think thats being tested regularly?16:02
sbezverknope, they do not start vm with cinder volumes I think16:03
sbezverkI think they use ephemeral disk16:03
kfox1111k. so it could be a regression.16:03
*** neilus has joined #openstack-kolla16:03
sbezverksorry, but where do you see in nova.conf v1?16:03
sbezverkhere is what you paster regarding to cinder16:03
sbezverk[cinder]16:03
sbezverkcatalog_info = volume:cinder:internalURL16:03
sbezverkdo not see any references to v116:04
kfox1111I think it needs to be: catalog_info = volumev2:cinderv2:internalURL16:04
kfox1111the v1 version doesn't have the v2 in it.16:04
kfox1111should match whats in the catalog.16:04
sbezverkhmm, never seen volume2 in this line16:05
sbezverkbut I guess I never paied too close attention16:05
kfox1111its a string to look up the specified endpoint in the catalog.16:06
kfox1111so the type/name/interface need to exist in the catalog.16:06
kfox1111and your patch changes the definition of volume:cinder:internalURL to be the v1 endpoint instead of the v2 endpoint.16:06
sbezverkcheck this out16:06
sbezverkhttps://bugs.launchpad.net/mos/+bug/150179816:06
openstackLaunchpad bug 1501798 in Mirantis OpenStack 9.x "nova is not using Cinder V2 API" [Wishlist,Fix released] - Assigned to MOS Nova (mos-nova)16:06
*** neilus has quit IRC16:08
kfox1111yeah.16:08
kfox1111I think kolla genconfig should be doing the same.16:08
kfox1111if forcing over to the v2 version works with your patch, I think it is something that should be fixed in kolla. hopefully there is still time.16:09
sbezverkyeah, they should not confiugre this value at all16:09
sbezverkbecause in nova code it defaults to v216:10
kfox1111that would work too.16:10
sbezverkhttps://review.openstack.org/#/c/124468/3/nova/volume/cinder.py16:10
kfox1111looking....16:10
kfox1111oh. kolla's probably overriding it because of the defaults publicURL, and thats a bit sketchy.16:11
sbezverkok in this case kolla should use v2 and internal url16:12
* kfox1111 nods16:14
sbezverkkfox1111: not related to current issue, do you know if ceph repo is still broken?16:15
sbezverk[ceph_deploy][ERROR ] RuntimeError: Failed to execute command: rpm -Uvh --replacepkgs http://ceph.com/rpm-jewel/el7/noarch/ceph-release-1-0.el7.noarch.rpm16:15
*** huikang has quit IRC16:15
*** huikang has joined #openstack-kolla16:16
kfox1111hmm. I usually use mirrors so I don't usually notice if it breaks.16:16
kfox1111can you wget it from all nodes?16:16
kfox1111the test just passed with your ps and the nova conf fix.16:18
kfox1111http://logs.openstack.org/19/386619/3/experimental/gate-kolla-kubernetes-deploy-ubuntu-binary-ceph-nv/bb42449/16:18
*** huikang has quit IRC16:20
sbezverkkfox1111: for demo I need to install single node cluster16:21
kfox1111so, kolla needs a fix and we can revert the ps back to version 2.16:22
sbezverkand that thing seems broken..16:22
kfox1111or we can merge ps 3 as is and just always override kolla.16:22
kfox1111sbezverk: too many conversation threads going on. are your last two statements related or is the broken comment about kolla genconfig?16:23
sbezverkoverride kolla for this looks as the easiest option16:23
kfox1111k. so lets merge #3 as is.16:23
kfox1111kolla's likely broken though.16:24
kfox1111so we should get it fixed there too, longer term.16:24
sbezverkkfox111: agree16:24
sbezverkkfox1111: back to my ceph issue16:27
sbezverkit is another issue, not related to recently discussed16:27
kfox1111k16:27
kfox1111whats up?16:27
sbezverkkfox1111: if ceph-deploy fails to install jewel, what are my options? install ceph manually?16:28
*** zigo has quit IRC16:28
sbezverkkfox1111: please check this link http://paste.openstack.org/show/585777/16:29
kfox1111looking...16:29
*** sdake has joined #openstack-kolla16:30
*** zigo has joined #openstack-kolla16:30
*** zigo is now known as Guest2399016:31
openstackgerritKevin Fox proposed openstack/kolla: Cinder endpoint wrong fix  https://review.openstack.org/38696516:31
*** janem has joined #openstack-kolla16:32
kfox1111rpms look to be here https://download.ceph.com/rpm-jewel/el7/x86_64/16:34
kfox1111and here: https://download.ceph.com/rpm-jewel/el7/noarch/16:34
kfox1111grep ceph-deploy/ceph-release from that last link. that will probably fix it.16:35
*** salv-orlando has joined #openstack-kolla16:37
sbezverkkfox1111: thanks pointing ceph-deploy to custom repo url helped16:39
kfox1111cool.16:40
*** salv-orlando has quit IRC16:42
*** v1k0d3n has joined #openstack-kolla16:43
*** huikang has joined #openstack-kolla16:49
*** huikang has quit IRC16:49
*** huikang has joined #openstack-kolla16:49
*** huikang has quit IRC16:50
*** huikang has joined #openstack-kolla16:50
*** spsingh has joined #openstack-kolla16:50
*** huikang has quit IRC16:53
*** SPSINGH__ has quit IRC16:54
*** SPSINGH__ has joined #openstack-kolla16:58
*** spsingh has quit IRC17:02
*** spsingh has joined #openstack-kolla17:04
*** SPSINGH__ has quit IRC17:04
*** tonanhngo has joined #openstack-kolla17:05
sbezverkkfox1111: are you still around? this ceph drivers me crazy.17:08
*** tonanhngo has quit IRC17:08
kfox1111yeah. whats up?17:12
*** shardy_afk has quit IRC17:12
sbezverkkfox1111: can you check this link17:17
sbezverkhttp://paste.openstack.org/show/585778/17:17
sbezverkon centos it tries to use service instead of systemctl to start monitor and obviously it fails17:17
sbezverkany ideas what I do wrong here?17:18
kfox1111looking...17:18
*** sdake_ has joined #openstack-kolla17:19
kfox1111looks like its an older ceph-deploy17:20
kfox1111what version is it? is i t the newest in https://download.ceph.com/rpm-jewel/el7/noarch/?17:20
kfox1111I think they switched to systemd only in infernalis. so, faily recently.17:21
kfox1111sdake_: https://review.openstack.org/#/c/386965/17:22
*** sdake has quit IRC17:22
sbezverkkfox1111: thanks17:24
kfox1111np17:26
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: Modular gate  https://review.openstack.org/38696617:27
*** ssurana has joined #openstack-kolla17:31
HyperJohnGraham_good morning all17:31
kfox1111morning17:32
sbezverkmorning17:32
HyperJohnGraham_sbezverk: is it time I try to install again or is the code still moving17:33
sdake_i recommend using kolla's ceph deploy mechanism - then we dont have to worry about version problems17:34
HyperJohnGraham_i am willing to reformat all my machines :)17:34
sbezverkkfox1111: can you ack that cinder ps?17:34
sbezverkHyperJohnGraham_: reformating seems a bit extreme17:34
HyperJohnGraham_several times ;)17:34
sbezverkHyperJohnGraham_: there are some steps, which I tested that really cleans kuberenetes and left overs from a server17:35
sdake_kfox1111 re the reviewy ou linked earlier, is that a change for newton where cinder volumes sare not backable by ceph?17:36
HyperJohnGraham_thats the issue i am fighting now on gateway.17:36
sbezverkbut your problem was inconsistency in your servers configuration17:36
HyperJohnGraham_yup17:36
sbezverkyou need to make sure that all servers are well prepared..17:36
HyperJohnGraham_formatted :)17:36
sbezverkalso you need to switch to btrfs for docker to have better performance..17:36
kfox1111sbezverk: will do.17:36
HyperJohnGraham_i did that17:37
kfox1111sdake_: the gate check for kolla-kubernetes turned it up. I think kolla's ceph support is broken on mitaka and newton's without that patch.17:38
HyperJohnGraham_how do i flush kube from my gateway machine thats running the external ceph17:38
HyperJohnGraham_or should i blow that away ?17:38
sdake_kfox1111 ceph works most definately17:38
sdake_kfox1111 but with cinder - not sure17:38
sdake_kfox1111 can you open a critical bug for this17:38
kfox1111yeah. it shows up when you go and try to detach a cinder volume from a vm thats backed by ceph.17:38
sdake_and change your commit message17:38
sdake_to include closes-bug:#xyz17:39
kfox1111k.17:39
sdake_thanks17:39
sdake_target it to ocata17:39
sdake_and link when done17:39
sdake_surprised we missed this17:39
kfox1111oh....17:40
kfox1111https://bugs.launchpad.net/kolla/+bug/162706117:40
openstackLaunchpad bug 1627061 in kolla "The url of Cinder endpoints isn't created correctly" [High,Fix released] - Assigned to caoyuan (cao-yuan)17:40
kfox1111sbezverk: ^---- thats why kolla-kubernetes had the wrong v1 url too. we coppied kolla initially.17:40
kfox1111sdake_: do you want a new bug or do we add to that bug?17:41
sdake_kfox1111 i'd like to see what that particular change was before we proceed17:41
kfox1111k17:41
sdake_yes new bug pleasse17:42
kfox1111so the bug is, kolla genconfig is telling nova to lookup the 'volume:cinder:internalURL' which is the v1 endpoint.17:42
sdake_this one is ceph specific17:42
kfox1111kolla was alwasy registering the v2 url, and nova expects the v2 url. so it worked.17:42
sdake_kfox1111 right - not alot of people use cinder17:42
sdake_no perviously we registered only the v1 endpoint17:42
*** SPSINGH__ has joined #openstack-kolla17:42
sdake_https://review.openstack.org/#/c/375775/2/ansible/roles/cinder/tasks/register.yml17:42
kfox1111previous to mitaka, it onlly supported v1.17:42
kfox1111I think.17:43
kfox1111so this was only broken in rc2 I think. so not very long.17:43
sdake_ya still needs a new bug - fix was released on that particular bug17:43
*** tonanhngo has joined #openstack-kolla17:44
kfox1111filing. sec.17:44
kfox1111sdake_: tag critical?17:45
sdake_priority critical17:45
kfox1111or some other way?17:45
kfox1111k17:45
sdake_thats the only way to get it in rc317:45
sdake_and if cinder is broken with ceph that seems pretty critical to me17:45
*** spsingh has quit IRC17:46
sdake_pretty much judgement call, it could be high or critical17:46
sdake_but since the work is already done - might as well include it as well :)17:46
kfox1111https://bugs.launchpad.net/kolla/+bug/163375817:46
openstackLaunchpad bug 1633758 in kolla "cinder wrong endpoint in nova" [Undecided,In progress] - Assigned to Kevin Fox (kevpn)17:46
kfox1111I can't set priority.17:46
openstackgerritKevin Fox proposed openstack/kolla: Cinder endpoint wrong fix  https://review.openstack.org/38696517:47
*** spsingh has joined #openstack-kolla17:47
sbezverkkfox1111: ceph really does not like me :-( http://paste.openstack.org/show/585780/17:48
sdake_kfox1111 you up for the cherypick backport as well - should be a click after the master change merges17:49
*** SPSINGH__ has quit IRC17:50
kfox1111sdake_: I can try. never done it before.17:50
sdake_kfox1111 its eze17:50
sdake_kfox1111 one click17:50
kfox1111k17:50
sdake_kfox1111 once it merges, let me know and i'll show you how to do it17:51
kfox1111sbezverk: trying to look... its taking a while to pring up the paste for some reason.17:51
kfox1111sdake_: kk17:51
kfox1111sbezverk: can you get to that paste? did the server go down?17:53
HyperJohnGraham_i think his paste crashed the server ;)17:54
kfox1111there we go. finally came up.17:54
sbezverkno, it is up ;-)17:56
kfox1111sbezverk: you said this is single node? any firewalls?17:57
sbezverkyes, single node17:57
sbezverklocal install no firewalld17:57
kfox1111hmm...17:58
kfox1111sbezverk: not sure off hand.18:03
kfox1111are you following the docs here: http://docs.ceph.com/docs/jewel/start/ ?18:03
*** spsingh has quit IRC18:06
kfox1111http://docs.ceph.com/docs/jewel/start/quick-ceph-deploy/#create-a-cluster mentions cleaning out if things go wrong.18:06
kfox1111things might have gotten out of sync when you started with an older ceph-deploy.18:06
*** spsingh has joined #openstack-kolla18:06
sbezverkkfox1111: ok will try to clean up18:07
sbezverkthanks18:07
kfox1111it looks like in the prep docs they mention creating a ceph-deploy user not named 'ceph' too. that seems new since the last time I tried a ceph-deploy.18:08
*** sdake_ has quit IRC18:11
*** SPSINGH__ has joined #openstack-kolla18:15
*** spsingh has quit IRC18:15
*** sdake has joined #openstack-kolla18:17
kfox1111sdake: when do you think the hub containers will get cut?18:20
kfox1111I'm going to propose a multinode job for kola-kubernetes and am wondering if I should slip in a 3.0 series as well.18:21
sdakekfox1111 coolsvap usually does them after we tag18:23
sdakekfox1111 i think he has a script for it - but not certain18:23
kfox1111soonish? a week or so?18:24
sdakedeadline is 18th for rc318:24
sdake3.0.0 will be tagged 2-3 days after that18:24
sdakefrom the rc3 tag18:24
sdakerc3 should be identical to 3.0.018:24
sdakethis gives us 2-3 days to make sure everything is a-ok and not DOA18:25
kfox1111ok. I'll do a 3.x series in a follow up ps then.18:25
sdakethat puts it right in the middle of summit18:25
sdakeas to when the images would be pushed ithink18:25
kfox1111heh. not the best time.18:26
sdakeya - i hate to be blunt, but everyone is relaly workign hard on 3.0.0 and making sure summit works well18:27
kfox1111oh well.18:27
sdakeso sort of busy to deal with other things18:27
sdakenot just me, but  eveyrone18:27
kfox1111yeah. I get that.18:27
sdakei have 3 presentations at summit18:27
sdakei am not prepared for any of them18:27
sdaketwo of my 3 computers are broken18:27
kfox1111ouch.18:27
sdakeand i have commandeered my wife's lifetop for summit18:27
sdakeand sending mine to repair in about 30 mins18:27
sdakeso getting what i've got backed up18:28
sdakeon top of that we have 10 critical bugs18:30
sdakei am wary of releasing 3.0.0 with them as is18:30
kfox1111bummer. :/18:30
sdakeyet progress is slow18:30
sdakealthough we have solutions for all of them18:30
sdakeexcept the tls horizon problem18:30
sdakewhich hasn't been confirmed18:30
kfox1111kind of the problem with hard release dates and volunteer workers. :/18:31
sdakeya18:31
kfox1111they don't always fit together.18:31
sdakeimpossible to do capacity planning18:31
* kfox1111 nods18:31
kfox1111our organization is heavily matrixed.  we also have dev and ops stuff going on by the same people.18:31
sdakeand alot of folks are working on master now rather then fixing 3..0.018:31
sdakeif we don't release a working 3.0.0 kolla will be in bad shape18:32
kfox1111and managers have a hard time understanding then why we can't give very firm dates. :/18:32
kfox1111"when the machine goes down, thats our top priority as an op. we can't predict when though"18:32
sdakecisco uses agile18:32
kfox1111yeah. we pretend to use agile.... but they don't get it really.18:32
sdakeso we have typical engineering teams, but not matrixed with operations18:32
kfox1111that would help.18:33
*** spsingh has joined #openstack-kolla18:37
*** SPSINGH__ has quit IRC18:37
*** sdake_ has joined #openstack-kolla18:38
*** salv-orlando has joined #openstack-kolla18:38
sdake_kfox see pm18:38
kfox1111k18:39
*** SPSINGH__ has joined #openstack-kolla18:39
*** sdake__ has joined #openstack-kolla18:41
sdake__kfox1111 see pm18:41
kfox1111your user names jumping around a lot.18:42
*** huikang has joined #openstack-kolla18:42
*** spsingh has quit IRC18:42
sdake__kfox1111 have two laptops connected atm18:42
sdake__and connected to vpn18:43
sdake__= 3 login ids :(18:43
*** salv-orlando has quit IRC18:44
*** sdake_ has quit IRC18:44
*** neilus has joined #openstack-kolla18:45
*** pbourke_ has quit IRC18:45
*** pbourke_ has joined #openstack-kolla18:45
*** sdake has quit IRC18:49
*** neilus has quit IRC18:50
*** jmccarthy has quit IRC18:56
*** jmccarthy has joined #openstack-kolla18:57
*** neilus has joined #openstack-kolla19:02
*** sdake has joined #openstack-kolla19:03
*** eaguilar has quit IRC19:05
*** msimonin has joined #openstack-kolla19:07
*** msimonin has quit IRC19:12
*** sdake has quit IRC19:18
*** janem has quit IRC19:25
*** sdake has joined #openstack-kolla19:27
*** msimonin has joined #openstack-kolla19:27
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: Modular gate  https://review.openstack.org/38696619:36
sdake__kfox1111 can you fix that commit you have outstanding for 3.0.0?19:37
sdake__with a closes-bug: #xyz19:37
kfox1111looking.19:38
sdake__the one about cinder and ceph19:38
kfox1111https://review.openstack.org/#/c/386965/ ?19:38
kfox1111missing space?19:39
sdake__and capital C and capital B19:39
sdake__Closes-Bug: #163375819:39
openstackbug 1633758 in kolla "cinder wrong endpoint in nova" [Critical,In progress] https://launchpad.net/bugs/1633758 - Assigned to Kevin Fox (kevpn)19:39
openstackgerritKevin Fox proposed openstack/kolla: Cinder endpoint wrong fix  https://review.openstack.org/38696519:39
*** salv-orlando has joined #openstack-kolla19:40
sdake__thanks :)19:40
kfox1111np. :)19:41
*** salv-orlando has quit IRC19:44
*** sdake has quit IRC19:45
*** huikang has quit IRC19:45
*** huikang has joined #openstack-kolla19:45
*** HyperJohnGraham_ has quit IRC19:47
*** haplo37 has quit IRC19:48
*** haplo37 has joined #openstack-kolla19:48
*** huikang has quit IRC19:50
*** jrollinhatin is now known as jroll19:51
*** sdake has joined #openstack-kolla19:58
*** huikang has joined #openstack-kolla19:59
*** sdake__ has quit IRC20:01
*** SPSINGH__ has quit IRC20:06
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: Modular gate  https://review.openstack.org/38696620:11
*** hogepodge_ has quit IRC20:28
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: Modular gate  https://review.openstack.org/38696620:33
*** salv-orlando has joined #openstack-kolla20:41
*** msimonin has left #openstack-kolla20:45
*** salv-orlando has quit IRC20:45
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: Modular gate  https://review.openstack.org/38696620:48
*** hogepodge has joined #openstack-kolla20:56
*** eaguilar has joined #openstack-kolla21:17
*** neilus has quit IRC21:25
*** janem has joined #openstack-kolla21:25
kfox1111there we go.. got this box to finally boot...21:32
kfox1111really weird bios...21:32
kfox1111early transition efi/bios.21:32
*** HyperJohnGraham_ has joined #openstack-kolla21:33
kfox1111vary rare that I have to change the defaults to even get a centos installer to boot.21:33
HyperJohnGraham_sbezverk: [#openstack-kolla] HyperJohnGraham_: there are some steps, which I tested that really cleans kuberenetes and left overs from a server21:37
*** huikang has quit IRC21:37
HyperJohnGraham_can you share ? :)21:37
*** huikang has joined #openstack-kolla21:38
*** janem has quit IRC21:41
*** salv-orlando has joined #openstack-kolla21:41
*** huikang has quit IRC21:42
*** salv-orlando has quit IRC21:46
*** eaguilar has quit IRC21:53
*** sdake has quit IRC21:58
*** sdake has joined #openstack-kolla22:00
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: Modular gate  https://review.openstack.org/38696622:02
*** neilus has joined #openstack-kolla22:10
*** neilus has quit IRC22:14
*** dave-mccowan has joined #openstack-kolla22:21
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: Modular gate  https://review.openstack.org/38696622:22
*** dave-mcc_ has joined #openstack-kolla22:31
*** ssurana has quit IRC22:32
*** dave-mccowan has quit IRC22:33
*** ssurana has joined #openstack-kolla22:34
*** sdake_ has joined #openstack-kolla22:35
*** ssurana has quit IRC22:37
*** sdake has quit IRC22:37
*** hogepodge has quit IRC22:40
*** hogepodge has joined #openstack-kolla22:40
*** salv-orlando has joined #openstack-kolla22:42
*** salv-orlando has quit IRC22:47
Pavoomg grrrr trying to get UEFI to PXE boot from anything linuz is a nightmare22:55
openstackgerritDave McCowan proposed openstack/kolla: OpenStack Services Should Use keystone_internal_url for auth  https://review.openstack.org/38697522:55
Pavolinux*22:55
*** janem has joined #openstack-kolla23:00
*** spsingh has joined #openstack-kolla23:03
*** sdake_ has quit IRC23:07
*** spsingh has quit IRC23:08
kfox1111Pavo: try cobbler.23:14
kfox1111it has booted uefi pxe stuff pretty easy for me.23:15
PavoI have been trying23:16
Pavocan you tell me how to get it to boot UEFI23:16
Pavobecause every guide I have read is for RHEL6.4 and its not working for CentOS 723:16
kfox1111I just downloaded the newest cobbler, and had it fetch the newest firmware.23:17
kfox1111getting the tftp logs or a tcpdump usually helps so you can watch what files it is trying to download.23:19
Pavousing cobbler get-loaders right23:19
kfox1111usualy gives you an indication if your missing some firmware files.23:19
kfox1111yeah.23:19
Pavoyeah when I select PXE on these HP servers it says NBR successful then go backs to boot menu23:20
Pavoover and over23:20
kfox1111yeah. I'd try looking at tcpdump or logs from the tftp server to see if what its trying to download is sent to it.23:21
sbezverkkfox1111: any chance you ack that ps, I would like to have it for demo cluster installation23:21
kfox1111k. was kind of hoping the kolla review would go through quickly, since sdake already acked it.23:24
kfox1111but I can.23:24
Pavokfox1111 sometime when you got some time I would love for you to walk me through cobbler for UEFI if you wouldn't mind23:26
kfox1111sbezverk: workflowed.23:26
kfox1111its been 6 months since I did it last. but I can try.23:27
Pavowould be awesome23:28
PavoI am about to rip my hair out what little hair I have left23:28
openstackgerritMerged openstack/kolla-kubernetes: Fixing cinder v1 and v2 endpoints  https://review.openstack.org/38661923:30
*** sdake has joined #openstack-kolla23:35
kfox1111Pavo: would relly recommand looking at the logs/tcpdump while trying. I think it will probably show you the issue.23:37
*** HyperJohnGraham_ has quit IRC23:39
Pavoand just doing a yum install cobbler will work with UEFI, no special settings I need?23:42
kfox1111I think it was just the yum install cobbler and the fetch firmware.23:43
*** salv-orlando has joined #openstack-kolla23:43
kfox1111but there might have been a flag on profile/server to force uefi too. don't remember if that was needed or not.23:43
Pavohmmm23:46
Pavothink you cand find that flag?23:46
Pavoor share your configs23:47
*** salv-orlando has quit IRC23:48
*** Jeffrey4l has joined #openstack-kolla23:49
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: Modular gate  https://review.openstack.org/38696623:51
kfox1111I'm pretty sure it didnt take effect.23:51
kfox1111the logs/tcpdump would really help.23:51
kfox1111that vast majority of issues I've had with pxe has been it looking for the wrong file or looking in the wrong place.23:52
kfox1111and the pxe clients never tell you what its trying to do. :/23:52

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!