Thursday, 2017-01-12

sdake_oslo.i18n doesn't work from the pip install00:00
*** g3ek has quit IRC00:00
sdake_might be an upper constraints problem00:00
*** saneax is now known as saneax-_-|AFK00:00
sdake_most people would never notice - however- I have a fresh box00:00
*** schwicht has joined #openstack-kolla00:01
sbezverksrwilkers_: ping00:02
*** sayantani01 has quit IRC00:07
sbezverksdake_: do you have by chance srwilkers_ email?00:07
sdake_sbezverk you can find it in gerrit00:07
sdake_moemnt00:07
sdake_https://review.openstack.org/#/admin/groups/1380,members00:08
kfox1111sdake_: worst worst case, just having a 3.0.2 for kolla-toolbox and kubernetes-entrypoint would really help.00:08
sdake_kfox1111 let me build and push and then we can see if they are busted via the gate ;)00:09
*** g3ek has joined #openstack-kolla00:09
kfox1111sdake_: sounds good.00:09
sdake_built binary - now buildilng source00:09
sdake_then i'll push them00:09
sdake_binary built correctly - yay :)00:09
kfox1111nice.00:11
sbezverksdake_: thanks00:12
*** Pavo has quit IRC00:14
*** Pavo has joined #openstack-kolla00:14
*** hfu has joined #openstack-kolla00:16
*** tonanhngo has quit IRC00:16
*** saneax-_-|AFK is now known as saneax00:16
*** schwicht has quit IRC00:19
*** schwicht has joined #openstack-kolla00:22
*** schwicht has quit IRC00:24
*** TxGirlGeek has quit IRC00:26
kfox1111bbiab00:26
*** zhurong has joined #openstack-kolla00:28
*** inc0 has quit IRC00:32
*** celo_r has quit IRC00:34
*** liyifeng has joined #openstack-kolla00:39
*** sdake has joined #openstack-kolla00:40
*** sdake_ has quit IRC00:41
*** duonghq_ has joined #openstack-kolla00:41
duonghq_morning guys00:41
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes: Glance service orchestration with entrypoint  https://review.openstack.org/41073600:43
sbezverkduonghq_: morning, had any luck with halcyon?00:43
*** sayantani01 has joined #openstack-kolla00:44
sdakeduonghq_ sbezverk has started work on operators00:44
sdakeduonghq_ if you are looking for some cool work - sbezverk can use a partner :)00:44
sdakekfox1111 images are built - pushing shortly00:44
duonghq_sdake, I saw that, trying to finish ceph microservices, and will move to operator soon00:45
duonghq_thank you sdake00:45
sdakeduonghq_ nice ;)00:45
duonghq_sbezverk,  I still keep the vagrant VM running00:45
duonghq_sdake, which image are you building? (base and type)00:45
*** hfu has quit IRC00:46
*** saneax is now known as saneax-_-|AFK00:46
*** hfu has joined #openstack-kolla00:47
*** sdake_ has joined #openstack-kolla00:47
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes: Mariadb service deployment  https://review.openstack.org/41223700:47
*** duonghq has left #openstack-kolla00:48
*** duonghq_ is now known as duonghq00:49
*** sdake has quit IRC00:49
*** schwicht has joined #openstack-kolla00:52
sdake_any idea who is building master on docker hub?00:54
*** schwicht has quit IRC00:55
*** schwicht has joined #openstack-kolla00:56
Pavodoes swift need anything special in kolla-ansible other than just enabling it in globals?00:57
sdake_pavo you have to create the disks00:59
sdake_or register them or something00:59
sdake_it is in the swift guide00:59
Pavoso can't use cepf as a backend for it00:59
sdake_nope00:59
Pavodamn00:59
sdake_kfox1111 - binary is half pushed01:00
Pavook can those disk be made on the storage node and not the compute node?01:01
sdake_pushing source centos next01:01
sdake_pavo - I htink they cna be anywhere a storage network has access to - so it doesn't matter in particular01:01
duonghqsdake_, how often do we push master image?01:01
sdake_but it may have to only be on the storage nodes - not certain01:01
sdake_duonghq no idea - I didn't think we were pushing master at all01:01
Pavook cool will make one of my ceph drives into swift01:02
duonghqso how do we deal with kolla-k8s?01:02
sdake_duonghq we are using 3.0.2 images at present01:02
sdake_rather 3.0.1 - soon to be 3.0.201:02
sdake_which I think is wrong - we need to be testing master of openstack images01:02
duonghqsdake_, agree on this point01:03
duonghqso I wonder how we achieve that01:03
sdake_we build in the gate instead of pulling from docker hub and push to a local registry in the gate01:03
*** PramodJ has joined #openstack-kolla01:03
sdake_this is called cross repo gating :)01:04
sdake_we need it done - haven't had time to address it and kfox1111 hasn't either01:04
duonghqWe can make a repo persistent in gate?01:04
sdake_no it dies every time - so it needs to be rebuilt every time01:04
duonghqI heard somebody talk about this, just wonder if we can make it persistent01:04
sdake_inc0 is working on a patch which puts a registry somewhere in the middle of that01:04
sdake_so itis persistent01:04
sdake_not sure how it works :)01:05
sdake_atm its based on puppet - so I am unable to spend 6-8 weeks to learn how to puppetize something like docker hub01:05
*** MasterOfBugs has quit IRC01:05
duonghqah, good news, building image is painful if we must do this every time01:05
sdake_full moon and friday the 13th all in one01:06
sdake_should be an interesting friday :)01:06
duonghqdidn't notice it01:07
*** pramodrj07 has quit IRC01:07
duonghqwhat should happen when both beautiful things happen?01:07
sdake_who knows01:08
*** PramodJ has quit IRC01:08
*** lrensing has quit IRC01:10
*** pramodrj07 has joined #openstack-kolla01:10
*** MasterOfBugs has joined #openstack-kolla01:10
*** pramodrj07 has quit IRC01:14
*** MasterOfBugs has quit IRC01:14
kfox1111back.01:15
duonghqsup kfox111101:15
kfox1111duonghq: hey. :)01:15
kfox1111sdake_: nice. :)01:15
*** zhubingbing has joined #openstack-kolla01:16
jemcevoyMech422 ... Question on your vlan setup with OVS.  Does that network config file you sent me in http://paste.openstack.org/show/594621/ run in one of the openvswitch containers or on the bare metal server?01:19
jemcevoyI ask because I'll need some of this setup before the openvswitch_db is created....01:21
*** sayantan_ has joined #openstack-kolla01:25
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes: Mariadb service deployment  https://review.openstack.org/41223701:27
*** Pavo has quit IRC01:27
*** mannamne has quit IRC01:27
*** sayantani01 has quit IRC01:27
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes: Glance service orchestration with entrypoint  https://review.openstack.org/41073601:32
*** jrobinson has joined #openstack-kolla01:32
sdake_kfox1111 - what work are we waiting on to tag?01:32
sdake_kfox1111 ubuntu binary images?01:33
kfox1111centos-binary and ubuntu-binary01:33
sdake_kfox1111 - I've got to jet soon and feed the kids- so won't be able to build ubuntu atleast unitil I get back01:33
sdake_kfox1111 i don't htink inc0 finished buildign ubuntu binary01:33
kfox1111inc0 said he's pushing ubuntu01:33
sdake_yes but we need it tonight to tag tomorrow right?01:33
kfox1111asap, yeah.01:34
kfox1111yay! docker pull kolla/centos-binary-kolla-toolbox:3.0.2 worked.01:34
kfox1111getting close now.01:35
sdake_kfox1111 can you run that through the gate01:35
sdake_kfox1111 and see if centos binary is working properly01:35
kfox1111did all upload?01:36
kfox1111rechecked.01:37
kfox1111https://review.openstack.org/#/c/419082/01:37
kfox1111once that goes green, we're good to go.01:37
sdake_kfox1111 - ubuntu binary is building now (on centos)01:38
sdake_it may not be totally working - so inc0 will have to push again01:38
sbezverksdake_: I believe inc0 was taking care of ubuntu01:39
sdake_sbezverk yes but it needs to be built now01:39
sdake_sbezverk and it hasn't been built01:39
sbezverksdake_: I see, did not know that it is so much trouble to start building01:39
sdake_i don't have an ubuntu system to build on01:39
sdake_so its a struggle ;)01:40
sbezverksdake_: hm I build ubuntu on centos machine, works like a charm01:40
sdake_ok dinner time - images are pushing as we speak01:40
sdake_i'll try to get on irc at dinner if possible01:40
kfox1111k. thx.01:42
sbezverkkfox1111: after renaming objects we have bunch of left overs folders01:43
*** lrensing has joined #openstack-kolla01:43
*** tovin07_ is now known as tovin0701:45
kfox1111yeah, side affect of generateing values files. :/01:45
kfox1111fresh checkout's easiest way to get rid of them. :/01:46
sbezverkkfox1111: but not after resolving merge conflicts.. :(01:47
kfox1111yeah. those are going to be a bit of a pain. dealing with that myself. :/01:48
kfox1111the neutron service package is quite a ways away from tunk now.01:48
*** Pavo has joined #openstack-kolla01:48
sbezverkkfox1111: yeah mariadb and glance are all screwed up too, I mean in services.. but well at least when they green they will get merged ;-)01:50
kfox1111yeah. the neutron one's looking really nice to config after the merge though.01:51
kfox1111went from a loop across all of the packages with config for each, to like one general config file. :)01:51
kfox1111will post in a few.01:51
sbezverkkfox1111: check it out tomorrow.. I am off for now, good night01:53
*** adrian_otto has quit IRC01:53
kfox1111sbezverk: ok. good nite.01:53
openstackgerritdesheng proposed openstack/kolla-ansible: Move etcd precheck into its own role  https://review.openstack.org/41923801:58
*** tovin07_ has joined #openstack-kolla02:01
*** pramodrj07 has joined #openstack-kolla02:02
*** MasterOfBugs has joined #openstack-kolla02:02
*** unicell has quit IRC02:02
openstackgerritKuo-tung Kao proposed openstack/kolla-ansible: provide a universal collectd template  https://review.openstack.org/41449902:06
*** schwicht has quit IRC02:15
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: neutron service package  https://review.openstack.org/41223902:15
*** schwicht has joined #openstack-kolla02:16
*** yingjun has joined #openstack-kolla02:21
kfox1111sdake_: this is new: http://logs.openstack.org/82/419082/4/check/gate-kolla-kubernetes-deploy-centos-binary-ceph-nv/ad1f5a5/console.html02:23
*** negronjl has quit IRC02:28
*** Pavo has quit IRC02:28
*** Pavo has joined #openstack-kolla02:29
zhubingbinghello02:33
zhubingbinghttps://review.openstack.org/#/c/418811/02:33
Pavohas anyone ever seen this before after a deployment https://www.dropbox.com/s/47z1k6nqvxkse66/Screenshot%202017-01-11%2021.38.38.png?dl=002:38
Pavogetting that when I try to login02:39
Pavo[Wed Jan 11 21:37:34.711409 2017] [:error] [pid 33] RuntimeError: Unable to create a new session key. It is likely that the cache is unavailable.02:42
Pavothats in horizon.log02:42
duonghqkfox1111, should kolla-kubernetes work with master image?02:48
kfox1111duonghq: should, but untested so far.02:51
duonghqkfox1111, copy02:51
*** schwicht_at_work has joined #openstack-kolla02:51
*** schwicht has quit IRC02:52
*** tovin07 has quit IRC02:52
*** schwicht_at_work has quit IRC02:56
*** schwicht has joined #openstack-kolla02:56
*** tovin07 has joined #openstack-kolla02:56
*** cliles has quit IRC03:02
*** goldyfruit has joined #openstack-kolla03:07
*** duonghq has quit IRC03:09
*** tovin07 has quit IRC03:09
*** sdake_ has quit IRC03:10
*** tovin07 has joined #openstack-kolla03:10
*** hieulq has quit IRC03:10
*** tovin07_ has quit IRC03:11
*** sdake has joined #openstack-kolla03:11
*** lrensing has quit IRC03:12
sdakekfox1111 my kernel oopsed during building03:13
*** cliles has joined #openstack-kolla03:13
*** mgiles has quit IRC03:14
*** sdake_ has joined #openstack-kolla03:15
*** adrian_otto has joined #openstack-kolla03:15
*** adrian_otto has joined #openstack-kolla03:16
*** ocaner has quit IRC03:16
*** sdake has quit IRC03:17
*** sdake_ has quit IRC03:19
*** sdake has joined #openstack-kolla03:22
*** adrian_otto has quit IRC03:22
*** duonghq has joined #openstack-kolla03:24
*** tovin07_ has joined #openstack-kolla03:24
*** hieulq has joined #openstack-kolla03:25
sdakeJeffrey4l_ ping03:29
sdakeJeffrey4l_ when your about - did you happen to test 3.0.2 with rbd?03:29
sdakerather with ceph03:29
sdakeJeffrey4l_ the k8s gate (which uses ceph unlike the rest of the gates) returns http://logs.openstack.org/82/419082/4/check/gate-kolla-kubernetes-deploy-centos-binary-ceph-nv/ad1f5a5/console.html#_2017-01-12_02_21_11_92976303:30
Mech422jemcevoy: those configs run on the bare metal - also, kolla will 'restart' ovs which caused me to lose config - I had to comment out the restart or something03:31
*** lrensing has joined #openstack-kolla03:34
*** zhurong has quit IRC03:39
*** zhubingbing has quit IRC03:39
sp_sdake: hi...03:42
*** schwicht_at_work has joined #openstack-kolla03:42
*** schwicht has quit IRC03:44
*** sayantan_ has quit IRC03:45
*** prameswar has joined #openstack-kolla03:45
sdakesup sp_03:45
sp_sdake: how kolla-k8s differ from Stackanetes ?03:46
Jeffrey4l_sdake, no. haven't test with ceph.03:46
sp_sdake: i am confused here.03:46
sdakeJeffrey4l_ can you do so - I think 3.0.2 is doa in multiple ways03:47
sdakesp_ confused how03:47
sp_sdake: Stacknetes also provide life cycle management using k8s as kolla-k8s does.03:48
Jeffrey4l_ok. will   try.03:48
sdakeJeffrey4l_ i just filed this bug: https://bugs.launchpad.net/kolla/+bug/165581703:49
openstackLaunchpad bug 1655817 in kolla "pip install kolla==3.0.2 followed by kolla-build fails" [Critical,Triaged]03:49
openstackgerritrajiv proposed openstack/kolla-kubernetes: Use upper-constraints for tox envs  https://review.openstack.org/41925303:49
sdakeJeffrey4l_ also if ceph is busted - that is another DOA point03:49
sdakeJeffrey4l_ also I created 3.0.3 :)03:50
Jeffrey4l_roger.03:50
Jeffrey4l_thansk.03:50
sdakeif 3.0.2 is really DOA we should strive to release newton update asap to fix the doa-ness03:50
Jeffrey4l_yes.03:50
sdakesp_ not sure where the question is coming from03:50
sdakesp_ i never said stackanetes didn't provide life cycle management03:51
sdakenor did I say it does - I have made no statements about stackanetes at all03:51
sp_sdake: of-course you didn't.03:53
sp_sdake thanks03:53
sdakesp_ happy to help03:53
sp_sdake: may be i should go little more dip down into this.03:53
sdakei dont know what dip down means sorry doesn't pass parser after 15 hours of workday :(03:54
*** qiliang27 has joined #openstack-kolla03:55
sp_sdake: I going through Stacknetes for better understanding and looking for exactly what is difference between stacknetes and kolla-k8s.03:55
sdakesp_ i have not examined stackanetes all that much03:56
*** sayantani01 has joined #openstack-kolla03:56
sdakeand by all that much I mean i looked at it once long ago and havne't really looked since03:56
sdakeif there isn't much delta - we should figure out how to fold the communities together03:57
sdakeif that is a thing that can be done :)03:57
*** Pavo has quit IRC03:57
sdake190 bugs in ocata 303:57
* sdake sadfaces03:57
sdakethe repo split thing really screwed up kolla's mojo with implementation03:58
sp_sdake: You are right. We should fold the communities if not much difference.03:59
sp_sdake: thanks03:59
*** schwicht has joined #openstack-kolla04:01
*** schwicht_at_work has quit IRC04:01
sdakekfox1111 all of the 3.0.2 images have been pushed04:10
duonghqkfox1111, around? does we add kubernetes-entrypoint just in ocata?04:11
sdakekfox1111 if they are broken - it may be that 3.0.2 is doa with regard to ceph04:11
*** goldyfruit has quit IRC04:11
*** Jeffrey4l__ has joined #openstack-kolla04:11
*** l4yerffeJ has joined #openstack-kolla04:11
*** l4yerffeJ_ has quit IRC04:13
*** Jeffrey4l_ has quit IRC04:14
*** mdnadeem has joined #openstack-kolla04:18
sdakeJeffrey4l__ http://tracker.ceph.com/issues/1628904:18
Jeffrey4l__yep. it looks the same.04:18
Jeffrey4l__what the ceph client version in k8s gate?04:19
*** jrobinson is now known as jrobinson-afk04:20
sdakeno clue04:22
*** adrian_otto has joined #openstack-kolla04:28
*** zhubingbing has joined #openstack-kolla04:28
*** adrian_otto has quit IRC04:28
*** skramaja has joined #openstack-kolla04:30
*** jtriley has joined #openstack-kolla04:32
Mech422Morning04:41
*** sp__ has joined #openstack-kolla04:41
*** jrobinson-afk is now known as jrobinson04:46
*** jtriley has quit IRC04:47
sdakeJeffrey4l__ this looks like the ceph client call:04:49
sdake2017-01-12 02:21:10.080105 | + kubectl exec ceph-admin -c main --namespace=kolla -- /bin/bash -c 'timeout 240s rbd create kollavolumes/mariadb --size 1024'04:49
sdakelooks like its calling ceph-admin04:49
Jeffrey4l__ceph-admin should be the resource/pod name.04:50
sdakeit is04:51
Jeffrey4l__it is calling "rbd create ... "04:51
Jeffrey4l__sdake, are the 3.0.2 image pushed into hub.docker.com already?04:53
sdakeJeffrey4l__ i've been working on it04:53
sdakeand that gate job was 3.0.204:53
sdakeiiuc04:53
sdakei have pushed centos binary04:53
sdakeand ubuntu binary04:53
sdakenow doing centos source04:53
Jeffrey4l__so k8s's gate is using the centos-binary u just  pushed?04:54
sdakeyup04:54
Jeffrey4l__ok. i am trying to set up a env based on centos-binary 3.0.204:54
*** unicell has joined #openstack-kolla04:55
sdakei used sudo pip install kolla==3.0.204:55
Jeffrey4l__to build the images?04:56
sdakeyup04:56
sdakethat is how you build images pushed to docker hub04:56
sdakethat way the tags are correct04:56
Jeffrey4l__roger. that is correct.04:56
*** saneax-_-|AFK is now known as saneax05:01
*** v1k0d3n has quit IRC05:07
*** v1k0d3n has joined #openstack-kolla05:07
*** v1k0d3n has quit IRC05:12
*** sayantani01 has quit IRC05:14
*** sayantani01 has joined #openstack-kolla05:14
*** sayantani01 has quit IRC05:19
*** schwicht has quit IRC05:21
*** sacharya has quit IRC05:25
*** salv-orlando has joined #openstack-kolla05:26
*** salv-orlando has quit IRC05:31
*** zhubingbing has quit IRC05:39
*** zhubingbing has joined #openstack-kolla05:39
*** cliles has quit IRC05:45
*** awidders_ has joined #openstack-kolla05:48
*** awiddersheim has quit IRC05:51
*** suka_ has joined #openstack-kolla06:09
*** unicell1 has joined #openstack-kolla06:16
*** unicell has quit IRC06:17
*** absubram has quit IRC06:17
suka_Can anyone help me solve the "pod (xxx) failed to fit in any node fit failure summary on nodes : MatchNodeSelector (1)" error on kubernetes during the final steps of kolla-kubernetes configuration/installation06:20
*** unicell1 has quit IRC06:21
*** unicell has joined #openstack-kolla06:21
*** mdnadeem has quit IRC06:22
*** zhurong has joined #openstack-kolla06:23
*** sacharya has joined #openstack-kolla06:25
openstackgerritHongZhi XU proposed openstack/kolla-ansible: Move swift precheck into its own role  https://review.openstack.org/41876206:25
*** skramaja has quit IRC06:26
*** skramaja has joined #openstack-kolla06:26
suka_Can anyone help me solve the "pod (xxx) failed to fit in any node fit failure summary on nodes : MatchNodeSelector (1)" error on kubernetes during the final steps of kolla-kubernetes configuration/installation06:27
jascott1suka_ it cant schedule the pod, maybe its asking for CPU shares or other resource thats not available?06:27
*** salv-orlando has joined #openstack-kolla06:28
jascott1is that the same error when node is missing required label? cant recall06:30
*** sacharya has quit IRC06:30
suka_@jascott1 I have made sure that I given enough resources. Kubernetes is able to recognize all the openstack services. The problem is just with scheduling the pods06:31
jascott1suka_ do you have labels on the nodes?06:31
suka_Yes. I do06:31
jascott1it should tell you exact resource problem, if its not in describe or logs maybe events06:32
*** salv-orlando has quit IRC06:32
*** mdnadeem has joined #openstack-kolla06:34
openstackgerritWei Cao proposed openstack/kolla-ansible: Fix cleanup-containers  https://review.openstack.org/41929606:35
*** suka_ has quit IRC06:36
*** msimonin has quit IRC06:39
*** msimonin has joined #openstack-kolla06:39
*** msimonin has quit IRC06:40
*** msimonin has joined #openstack-kolla06:40
*** msimonin has quit IRC06:41
*** msimonin has joined #openstack-kolla06:41
*** msimonin has quit IRC06:42
*** msimonin has joined #openstack-kolla06:42
*** msimonin has quit IRC06:42
*** msimonin has joined #openstack-kolla06:43
*** msimonin has quit IRC06:43
*** satyar has joined #openstack-kolla06:46
*** salv-orlando has joined #openstack-kolla06:57
*** lrensing has quit IRC06:59
openstackgerritWei Cao proposed openstack/kolla-ansible: Change 1-subnet to public1-subnet in tools/init-runonce  https://review.openstack.org/41930907:13
*** duonghq has quit IRC07:16
*** mdnadeem has quit IRC07:24
*** mdnadeem has joined #openstack-kolla07:25
*** tovin07 has quit IRC07:28
*** logan- has quit IRC07:47
*** logan- has joined #openstack-kolla07:56
*** matrohon has joined #openstack-kolla07:59
*** sacharya has joined #openstack-kolla08:01
*** haplo37_ has quit IRC08:05
*** sacharya has quit IRC08:06
*** haplo37_ has joined #openstack-kolla08:09
*** jrobinson has quit IRC08:12
*** PramodJ has joined #openstack-kolla08:15
*** pramodrj07 has quit IRC08:18
*** MasterOfBugs has quit IRC08:18
*** MasterOfBugs has joined #openstack-kolla08:18
openstackgerritSurya Prakash Singh proposed openstack/kolla-ansible: Docs update related to hostname resolution in RabbitMQ  https://review.openstack.org/41825208:22
*** strigazi_AFK is now known as strigazi08:28
*** mgoddard has joined #openstack-kolla08:32
*** caowei has quit IRC08:39
*** salv-orl_ has joined #openstack-kolla08:43
*** salv-orlando has quit IRC08:43
*** egonzalez has joined #openstack-kolla08:44
egonzalezmorning08:44
*** DanyC has joined #openstack-kolla08:46
*** DanyC has quit IRC08:50
*** skramaja has quit IRC08:52
Jeffrey4l__sdake, around?08:53
*** Serlex has joined #openstack-kolla08:55
Jeffrey4l__sdake, i tested 3.0.2 with ceph, it works as expected.08:57
*** athomas has joined #openstack-kolla08:58
openstackgerritjianyi proposed openstack/kolla-ansible: Add zun ansible role  https://review.openstack.org/41617908:58
*** yingjun has quit IRC09:00
*** david-lyle has quit IRC09:01
*** sacharya has joined #openstack-kolla09:01
*** mgoddard has quit IRC09:02
*** prameswar has quit IRC09:03
openstackgerritjianyi proposed openstack/kolla-ansible: Add zun ansible role  https://review.openstack.org/41617909:04
*** sacharya has quit IRC09:06
openstackgerritSurya Prakash Singh proposed openstack/kolla: Heat configuration param change for heat-api section  https://review.openstack.org/41933009:10
*** DanyC has joined #openstack-kolla09:11
*** PramodJ has quit IRC09:13
*** MasterOfBugs has quit IRC09:13
*** DanyC has quit IRC09:15
*** DanyC has joined #openstack-kolla09:16
openstackgerritSurya Prakash Singh proposed openstack/kolla: Heat configuration param change for heat-api section  https://review.openstack.org/41933009:16
openstackgerritWei Cao proposed openstack/kolla-ansible: Fix cleanup-containers  https://review.openstack.org/41929609:18
*** DanyC has quit IRC09:21
*** david-lyle has joined #openstack-kolla09:24
satyarHi jeffrey4l__09:24
*** shardy has joined #openstack-kolla09:25
*** salv-orl_ has quit IRC09:26
*** gfidente has joined #openstack-kolla09:27
*** gfidente has joined #openstack-kolla09:27
*** caowei has joined #openstack-kolla09:30
*** qiliang28 has joined #openstack-kolla09:37
openstackgerritOpenStack Proposal Bot proposed openstack/kolla: Updated from global requirements  https://review.openstack.org/41934109:37
openstackgerritOpenStack Proposal Bot proposed openstack/kolla-ansible: Updated from global requirements  https://review.openstack.org/41934209:37
openstackgerritOpenStack Proposal Bot proposed openstack/kolla-kubernetes: Updated from global requirements  https://review.openstack.org/41934309:37
*** qiliang27 has quit IRC09:37
*** qiliang28 is now known as qiliang2709:37
*** zhurong has quit IRC09:44
*** mgoddard has joined #openstack-kolla09:44
*** hfu has quit IRC09:48
openstackgerritSurya Prakash Singh proposed openstack/kolla: Heat configuration param change for heat-api section  https://review.openstack.org/41933009:53
*** qiliang27 has quit IRC09:53
*** mannamne has joined #openstack-kolla09:57
*** prameswar has joined #openstack-kolla10:04
*** nwik has quit IRC10:12
Jeffrey4l__sup satyar,10:21
*** ccesario has joined #openstack-kolla10:22
openstackgerritWei Cao proposed openstack/kolla-ansible: Fix cleanup-containers  https://review.openstack.org/41929610:22
*** zhurong has joined #openstack-kolla10:25
openstackgerritEduardo Gonzalez proposed openstack/kolla-ansible: Ansible config for nova-serialproxy console  https://review.openstack.org/41587010:27
*** satyar has quit IRC10:35
sp_pbourke: it would be great if you review it again https://review.openstack.org/#/c/411313/10:36
*** caowei has quit IRC10:41
*** gfidente has quit IRC10:42
sp_pbourke: thanks for review10:43
*** JoseMello has joined #openstack-kolla10:43
pbourkesp_: no problem10:44
*** pbourke has quit IRC10:45
*** pbourke has joined #openstack-kolla10:45
openstackgerritMerged openstack/kolla-ansible: Move etcd precheck into its own role  https://review.openstack.org/41606310:47
sp_egonzalez: pbourke: Jeffrey4l__: zhubingbing: I have incorporated your comments please review again https://review.openstack.org/#/c/418252/10:48
sp_need to focus on BP too10:49
zhubingbingok10:49
zhubingbingfix now10:49
sp_yes10:49
zhubingbingdone10:50
*** shardy has quit IRC10:51
openstackgerritcaoyuan proposed openstack/kolla-ansible: Move kibana precheck into its own role  https://review.openstack.org/41883410:51
Jeffrey4l__sp_, nit :)10:53
openstackgerritEduardo Gonzalez proposed openstack/kolla-ansible: Unify jinja syntax  https://review.openstack.org/41941510:53
openstackgerritMerged openstack/kolla-ansible: Move murano precheck into its own role  https://review.openstack.org/41511310:53
*** saneax is now known as saneax-_-|AFK10:55
sp_Jeffrey4l__: it was a bit confusing that time to decide the exact section for this. After the egonzalez comment i moved it to that section.10:56
sp_Jeffrey4l__: though np, I will create new section and will move it to that section10:57
egonzalezpbourke: were you yesterday in the meeting?10:57
Jeffrey4l__but full qulified domain name is not related rabbitmq hostname issue.10:57
Jeffrey4l__they are totally different. fqdn is not required in most of cause.10:58
Jeffrey4l__but rabbitmq hostname is essential .10:58
*** Jeffrey4l__ is now known as Jeffrey4l10:58
Jeffrey4legonzalez, pbourke could u review this https://review.openstack.org/41550411:00
*** sacharya has joined #openstack-kolla11:02
pbourkeegonzalez: yes11:03
pbourkeegonzalez: but wasn't paying close attention11:03
*** qiliang27 has joined #openstack-kolla11:03
openstackgerritJeffrey Zhang proposed openstack/kolla: Bump openstack_release to 3.0.3 in group_vars  https://review.openstack.org/41941711:03
egonzalezpbourke: just was no make you know ptg brainstorming in case you missed https://etherpad.openstack.org/p/kolla-ocata-ptg11:04
pbourkeegonzalez: ah thanks11:04
*** mgoddard has quit IRC11:05
openstackgerritSurya Prakash Singh proposed openstack/kolla-ansible: Docs update related to hostname resolution in RabbitMQ  https://review.openstack.org/41825211:05
openstackgerritMerged openstack/kolla: Fix telegraf contaiers  https://review.openstack.org/41881111:06
sp_Jeffrey4l: done11:06
Jeffrey4lLGTM. thanks.11:06
*** sacharya has quit IRC11:07
*** qiliang27 has quit IRC11:09
*** qiliang27 has joined #openstack-kolla11:10
*** shardy has joined #openstack-kolla11:14
*** zhurong has quit IRC11:15
openstackgerritMerged openstack/kolla: Cover environment for kolla  https://review.openstack.org/41131311:16
pbourkeJeffrey4l: will tryi tout soon11:19
*** jmccarthy has joined #openstack-kolla11:19
pbourke*it out11:19
pbourkeJeffrey4l: will it improve error reporting or is it more to neaten the code11:20
*** gfidente has joined #openstack-kolla11:20
*** gfidente has joined #openstack-kolla11:20
Jeffrey4lpbourke, it is more to neaten the code11:22
pbourkeok11:22
pbourkeoptimise is probably not accurate in the commit message then11:23
pbourkebut its fine11:23
openstackgerritshaofeng cheng proposed openstack/kolla-ansible: Update README.rst in kolla-ansible  https://review.openstack.org/41052811:23
sp_egonzalez: pbourke incorporated Jeffrey4 comments too. would be great if you review now. https://review.openstack.org/41825211:24
Jeffrey4lpbourke, hrm. which world can explain this more accuracy? refactor?11:27
openstackgerritMerged openstack/kolla-ansible: Update kolla-ansible links on README  https://review.openstack.org/41846711:27
pbourkeJeffrey4l: refactor is the correct term11:28
Jeffrey4lok. i can fix it if necessary.11:29
egonzalezJeffrey4l: issues in newton gates are dependent of this change? https://review.openstack.org/#/c/419417/11:30
Jeffrey4legonzalez, yep.11:30
Jeffrey4lthis patch should fix the gate.11:30
egonzalezOK, thanks11:31
openstackgerritMerged openstack/kolla-ansible: Docs update related to hostname resolution in RabbitMQ  https://review.openstack.org/41825211:31
*** zhubingbing has quit IRC11:32
sp_Jeffrey4l: pbourke egonzalez thanks for review :)11:34
*** sayantani01 has joined #openstack-kolla11:35
*** sayantani01 has quit IRC11:40
*** infinitydon has joined #openstack-kolla11:45
*** sp__ has quit IRC11:47
*** Pavo has joined #openstack-kolla11:52
berendt[Openstack-operators] [kolla] deprecation/removal of Debian images: http://lists.openstack.org/pipermail/openstack-operators/2017-January/012427.html11:52
*** shardy has quit IRC11:55
*** gfidente has quit IRC11:55
*** shardy has joined #openstack-kolla11:56
*** gfidente has joined #openstack-kolla11:56
*** gfidente has joined #openstack-kolla11:56
*** masber has quit IRC11:56
*** saneax-_-|AFK is now known as saneax11:58
*** gfidente has quit IRC12:01
*** gfidente has joined #openstack-kolla12:02
*** gfidente has quit IRC12:02
pbourkeJeffrey4l: im seeing passwords being output from ansible with the refactor where they werent before12:02
pbourkeJeffrey4l: e.g http://paste.openstack.org/show/594682/12:03
*** dave-mccowan has joined #openstack-kolla12:13
*** msimonin has joined #openstack-kolla12:18
*** mdnadeem has quit IRC12:19
openstackgerritjianyi proposed openstack/kolla-ansible: Add zun ansible role  https://review.openstack.org/41617912:22
sdakemorning folks12:23
sdakeJeffrey4l any update on ceph 3.0.2?12:23
sdakeegonzalez - grats dude I noticed on linked in you got a new gig at ibm :)12:24
*** sdake_ has joined #openstack-kolla12:27
*** ipsecguy has quit IRC12:29
*** sdake has quit IRC12:29
*** bjolo has joined #openstack-kolla12:33
bjologood afternoon12:33
bjolocan someone please help me check out one thing. Trying to determine if it is a bug or corrupt database. please run command: openstack role assignment list --names12:34
bjolopaste the output12:35
*** caowei has joined #openstack-kolla12:35
*** pomac has joined #openstack-kolla12:37
pomacWe changed, default domain id in the keystone configuration and we want to know if we have a serius bugfix or just something that went wrong with our installation12:38
pomac(continuation of bjolo's message above)12:38
*** sdake_ has quit IRC12:43
egonzalezbjolo pomac http://paste.openstack.org/show/594686/12:44
egonzalezjust glance and keystone deployed12:44
bjolotnx egonzalez12:46
egonzalezbjolo: what happened?12:47
pomacWe got domain_id == None12:47
pomacso eventually it fails with NoneType has no len()12:47
openstackgerritSurya Prakash Singh proposed openstack/kolla-ansible: Cover environment for kolla-ansible  https://review.openstack.org/41945012:47
pomacegonzalez: so question is, did we miss something, or is it a real bug12:48
bjolohistory is that this used to work, but to enable v2 authentications we set default_domain_id = foobar in keystone.conf12:48
bjolothat seems to have broken the database in some way12:48
*** sdake has joined #openstack-kolla12:48
*** zhubingbing has joined #openstack-kolla12:49
sdakesup bjolo12:53
bjolohi sdake12:54
bjolohows it going12:54
sdakebusy12:54
bjoloup late or early?12:54
sdake6am12:54
sdakealways up by 5:30-6am12:55
sdakewent to bed around 9 or 10pm12:55
sdakeso up at a normal time I think is the correct answer :012:55
sdakealthough it may be early for others12:55
*** schwicht has joined #openstack-kolla12:55
sdakei have two kids in school - and my daughter starts at 7:3012:55
sdakeso need to wake up by 6 to get the house rolling12:56
bjolopretty much the same for me12:58
*** dgonzalez has quit IRC12:58
bjoloegonzalez, can i ask you to run one more thing?12:58
egonzalezbjolo: yes12:58
bjolomysql12:58
bjolokeystone db12:58
*** liyifeng has quit IRC12:58
bjoloselect * from project;12:58
pomacegonzalez: basically, we don't know where this NULL/None value comes form12:59
*** mdnadeem has joined #openstack-kolla13:00
egonzalezhttp://paste.openstack.org/show/594690/13:01
*** sacharya has joined #openstack-kolla13:03
*** salv-orlando has joined #openstack-kolla13:04
*** dgonzalez has joined #openstack-kolla13:05
*** sacharya has quit IRC13:08
mliimagood morning awesome guys13:08
mliimaand girls13:08
bjoloegonzalez, tnx. looks the same as ours tho13:08
mliima;)13:08
*** yangzb09 has joined #openstack-kolla13:09
*** mdnadeem has quit IRC13:09
*** gfidente has joined #openstack-kolla13:11
*** mliima_ has joined #openstack-kolla13:14
openstackgerritDan Prince proposed openstack/kolla: Add Apache packages to nova-api container  https://review.openstack.org/41813113:17
*** mliima has quit IRC13:17
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes: Glance service orchestration with entrypoint  https://review.openstack.org/41073613:18
*** caowei has quit IRC13:21
egonzalezbjolo pomac all my keystone db http://paste.openstack.org/show/594694/13:21
openstackgerritDan Prince proposed openstack/kolla: Add Apache packages to cinder-api container  https://review.openstack.org/41907613:22
egonzalezservices enabled glance heat and keystone only13:22
openstackgerritDan Prince proposed openstack/kolla: Add Apache packages to nova-api container  https://review.openstack.org/41813113:22
*** ipsecguy has joined #openstack-kolla13:23
pomacegonzalez: thanks, have you ever been looking at the keystone code? =)13:23
egonzaleztook a look, keystone removed part of v2.0 code as part of deprecation13:24
pomacegonzalez: i think i need to have someone explain parts of it to me - the domain_id bit that we're having problems with is actually explicitly set to "None"13:25
openstackgerritDan Prince proposed openstack/kolla: Add Apache packages to nova-api container  https://review.openstack.org/41813113:26
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes: Mariadb service deployment  https://review.openstack.org/41223713:26
openstackgerritDan Prince proposed openstack/kolla: Add Apache packages to cinder-api container  https://review.openstack.org/41907613:26
pomacegonzalez: keystone/resource/backends/base.py, get_project (line 564) ends with calling get_project_from_domain --- And that is funny! since it actually sets domain_id to None and this is before we want to use it in our lookup13:27
pomacegonzalez: if you could play with adding domains and see if it starts to fail, or just tell me that my reading of the code is wrong - hold on, will provide a stacktrace13:28
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes: Mariadb service deployment  https://review.openstack.org/41223713:28
pomacegonzalez: http://paste.openstack.org/show/594697/13:29
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes: Glance service orchestration with entrypoint  https://review.openstack.org/41073613:30
*** schwicht has quit IRC13:30
pomacegonzalez: if you don't have the time/energy then don't worry, will be pushing a real bug on it as well - we're deploying a new environment to test with currently13:32
*** yangzb09 has quit IRC13:32
sdakesup mliima_13:38
*** salv-orlando has quit IRC13:39
sdakesay egonzalez - I think you missed my last msg13:39
sdakeegonzalez - grants on the new gig at IBM :)13:39
openstackgerritMerged openstack/kolla: Bump openstack_release to 3.0.3 in group_vars  https://review.openstack.org/41941713:41
openstackgerritMerged openstack/kolla: Updated from global requirements  https://review.openstack.org/41934113:41
egonzalezsdake: thanks, was at lunch time then13:42
egonzalezalso moved to Ireland (someday i'll learn english)13:42
sdakeegonzalez I recall when you started on kolla you weren't really allowed to work on openstack during work hours13:42
sdakehas that changed?13:42
sdakeegonzalez - you seem to know english to me?  or you mean spoken english13:42
egonzalezspoken and better gramar13:43
sdakeegonzalez my spanish is terrible :)13:43
sdakeegonzalez even living in AZ on the border with mexico, I struggle to compose basic 2 word statements :)13:43
sdakeyour english is better then my spanish :)13:44
egonzalezyeah, In my previous company i wasn't able to work, now I can do when Im not too busy13:44
mliima_improve my english is one goal for this year13:46
mliima_:)13:46
pomacegonzalez: heh, have fun in Ireland, I really like it everytime i go there to visit family13:47
*** srwilkers has joined #openstack-kolla13:48
*** absubram has joined #openstack-kolla13:51
egonzalezzhubingbing: were you able to fix mariadb logs?13:53
zhubingbingnp13:54
*** schwicht has joined #openstack-kolla13:54
zhubingbingegonzalez  i fix it now, wait13:54
openstackgerritMerged openstack/kolla: Add Apache packages to ironic-pxe container  https://review.openstack.org/41909213:54
sdakeegonzalez i have been to dublin before - nice city13:55
sdakeJeffrey4l - any word on the ceph gate problem?13:55
sdakeI am taking pto today and tomorrow and wanted to get kolla-kubernetes 0.4.0 sorted13:56
sdakefor tagging13:56
sdakewe need a ceph gate for kolla-ansible as well - so that we catch these problems up front13:56
sdake(if anyone looking for some challenging work to do beyond the 190 bugs in the kolla bug tracker)13:56
egonzalezyeah, the only thing I miss from spain is food13:57
sdakei have been to barcelona13:57
sdakevery nice13:57
sdakei could definately have a condo there someday if I could afford it ;)13:57
sdakeand if they permit ownership from other countries13:57
*** lamt has joined #openstack-kolla13:58
sdakewhat was crazy about ireland is everyone there doesn't steal and stuff13:58
sdakeI put my bags in a truck13:58
sdakerather in a bus13:58
sdakeand like 15 or 20 people got off13:58
sdakeand I was worried my bags would be stolen13:58
sdakeand when I got done, the bags were still there13:58
sdakethat isn't something that would work in the US13:58
sdakethe 15-20 people  got off at different stops13:59
sdakereal eye opener about the irish culture13:59
*** v1k0d3n has joined #openstack-kolla13:59
*** absubram has quit IRC14:00
*** absubram has joined #openstack-kolla14:00
*** sdake_ has joined #openstack-kolla14:01
*** schwicht has quit IRC14:02
srwilkersyeah, i could see living in barcelona -- really enjoyed it14:03
*** rhallisey has joined #openstack-kolla14:03
*** schwicht has joined #openstack-kolla14:04
*** sdake has quit IRC14:05
*** shardy has quit IRC14:05
*** shardy has joined #openstack-kolla14:06
*** srwilkers is now known as srwilkers_coffee14:09
*** saneax is now known as saneax-_-|AFK14:16
pbourkesdake_: they could very easily be stolen ;)14:24
pbourkesdake_: i would be careful about that next time14:24
pbourkeits not too likely but about as likely as the US I imagine14:25
*** schwicht has quit IRC14:25
Jeffrey4lsdake_, i tested kolla 3.0.2 docker images with ceph enabled, it works.14:26
Jeffrey4lbut no idea why kolla-k8s failed.14:26
*** satyar has joined #openstack-kolla14:30
*** jtriley has joined #openstack-kolla14:35
*** schwicht has joined #openstack-kolla14:38
*** salv-orlando has joined #openstack-kolla14:39
openstackgerritcaoyuan proposed openstack/kolla-ansible: Move mongodb precheck into its own role  https://review.openstack.org/41606014:41
*** salv-orlando has quit IRC14:41
*** salv-orlando has joined #openstack-kolla14:41
openstackgerritcaoyuan proposed openstack/kolla-ansible: Move rabbitmq precheck into its own role  https://review.openstack.org/41773114:43
openstackgerritcaoyuan proposed openstack/kolla-ansible: Move kuryr precheck into its own role  https://review.openstack.org/41902214:46
*** srwilkers_coffee is now known as srwilkers14:46
openstackgerritcaoyuan proposed openstack/kolla-ansible: Move Influxdb precheck into its own role  https://review.openstack.org/41887514:54
*** yingjun has joined #openstack-kolla14:56
*** lrensing has joined #openstack-kolla14:57
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes: Glance service orchestration with entrypoint  https://review.openstack.org/41073614:59
sbezverkJeffrey4l: which failure you are refering to? Do you have any logs/links?15:00
Jeffrey4lsbezverk, this one https://review.openstack.org/#/c/419082/15:01
egonzalezJeffrey4l: remeber libvirt 2.0 issue in gates with xen nodes? Not only affect xen, i'm facing the same with qemu/kvm15:01
Jeffrey4lseem k8s do not work with kolla 3.0.215:01
Jeffrey4legonzalez, yep. this issue happen in different cause.15:01
*** lamt has quit IRC15:02
Jeffrey4li saw this issue when running qemu in kvm too.15:02
egonzalezthough was only with xen15:02
Jeffrey4lit seem related to how qemu ( without kvm ) emulate the cpu.15:03
*** lamt has joined #openstack-kolla15:03
*** sacharya has joined #openstack-kolla15:04
sbezverkJeffrey4l: that must be related to packaging for ceph15:04
Jeffrey4lsbezverk, i tested 3.0.2 with kolla-ansible. it works.15:04
sbezverkhttp://tracker.ceph.com/issues/1628915:05
openstackgerritOpenStack Proposal Bot proposed openstack/kolla-ansible: Updated from global requirements  https://review.openstack.org/41934215:05
sbezverkJeffrey4l: I am not say it is the image problem, I know kfox1111 did some stuff for ceph client in the container used at the gate..15:06
Jeffrey4lsbezverk, seems need add some parameter when calling `create command`15:06
Jeffrey4lok. sound should make some change in the kolla-k8s gate script.15:07
openstackgerritcaoyuan proposed openstack/kolla-ansible: Add elasticsearch precheck into its own role  https://review.openstack.org/41950715:07
*** absubram has quit IRC15:08
*** lrensing has quit IRC15:08
*** sacharya has quit IRC15:09
*** lrensing has joined #openstack-kolla15:09
openstackgerritcaoyuan proposed openstack/kolla-ansible: Move elasticsearch precheck into its own role  https://review.openstack.org/41950715:09
*** adrian_otto has joined #openstack-kolla15:09
*** yingjun has quit IRC15:11
*** yingjun has joined #openstack-kolla15:11
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes: Mariadb service deployment  https://review.openstack.org/41223715:16
openstackgerritcaoyuan proposed openstack/kolla-ansible: Move rally precheck into its own role  https://review.openstack.org/41951315:19
sdake_sbezverk right - isaid it was a client problem15:19
sdake_sbezverk - but I could be wrong as I don't know precisely how the k8s gate woroks - just a curosory understanding of it15:19
sdake_sbezverk this is why i asked Jeffrey4l to test kolla-kubernetes with ceph - to validate 3.0.2 wasn't DOA15:19
sdake_the i18n problem can be easily fixed15:20
sdake_a ceph problem - harder to fix15:20
sdake_btw folks I am on PTO today and tomororw but around a bit15:20
sdake_if you ping me I may respond15:20
openstackgerritEdmund Rhudy proposed openstack/kolla: Enable no-Internet build for most Ubuntu binary containers  https://review.openstack.org/41463915:21
Pavoanyone know why ceph-osd would exit with Container exited with non-zero return code15:22
PavoI am getting errors during TASK [ceph : Bootstrapping Ceph OSDs]15:23
*** yingjun has quit IRC15:23
PavoI have labeled all my disk for ceph15:24
openstackgerritcaoyuan proposed openstack/kolla-ansible: Move telegraf precheck into its own role  https://review.openstack.org/41952015:24
Pavousing parted /dev/sdb -s -- mklabel gpt mkpart KOLLA_CEPH_OSD_BOOTSTRAP 1 -115:24
Pavobut during deployment I am getting failed: [compute2] (item=(0, {u'fs_uuid': u'', u'journal_device': u'/dev/sdc', u'journal': u'/dev/sdc2', u'partition': u'/dev/sdc1', u'partition_num': u'1', u'journal_num': 2, u'fs_label': u'', u'device': u'/dev/sdc', u'external_journal': False})) => {"changed": true, "failed": true, "item": [0, {"device": "/dev/sdc", "external_journal": false, "fs_label": "", "fs_uuid": "", "journal":15:24
Pavo "/dev/sdc2", "journal_device": "/dev/sdc", "journal_num": 2, "partition": "/dev/sdc1", "partition_num": "1"}], "msg": "Container exited with non-zero return code"}15:24
Pavoonly on compute nodes15:24
Pavoanyone have a suggestion?15:25
*** adrian_otto has quit IRC15:27
*** adrian_otto1 has joined #openstack-kolla15:27
*** yingjun has joined #openstack-kolla15:29
*** yingjun has joined #openstack-kolla15:29
*** adrian_otto1 has quit IRC15:30
*** mgiles has joined #openstack-kolla15:32
*** infinitydon has quit IRC15:32
*** inc0 has joined #openstack-kolla15:32
inc0good morning15:32
Pavoinc0 morning15:33
inc0erhudy, I saw your email, I'll look into it15:34
erhudyinc0: thanks, let me know what i can do to help15:34
Pavoany reason why bootstrap ceph osd wouldn't be able to partition drives that are labeled during deployment?15:34
inc0also you and Pavo should talk:) both of you are deploying hyper security sensitive OS15:34
inc0Pavo, it does partition them15:34
Pavoit failing for me15:35
inc0hmm15:35
Pavoonly on my computes though15:35
erhudydo you have leftover ceph structures on the drives from before?15:35
inc0interesting, with what error?15:35
PavoI did a fdisk and wiped them all out prior15:35
Pavobut during deployment I am getting failed: [compute2] (item=(0, {u'fs_uuid': u'', u'journal_device': u'/dev/sdc', u'journal': u'/dev/sdc2', u'partition': u'/dev/sdc1', u'partition_num': u'1', u'journal_num': 2, u'fs_label': u'', u'device': u'/dev/sdc', u'external_journal': False})) => {"changed": true, "failed": true, "item": [0, {"device": "/dev/sdc", "external_journal": false, "fs_label": "", "fs_uuid": "", "journal":15:35
Pavo "/dev/sdc2", "journal_device": "/dev/sdc", "journal_num": 2, "partition": "/dev/sdc1", "partition_num": "1"}], "msg": "Container exited with non-zero return code"}15:35
erhudyi've had problems with ceph trying to re-deploy to drives that were previously used for ceph, sgdisk -Z to destroy the GPT structures and a reboot so the kernel would forget the partition table was necessary15:35
Pavook will try that15:36
Pavoonly thing I have changed now using btrfs instead of lvm for host OS15:36
Pavowhats the cmd erhudy?15:37
erhudya bold move15:37
erhudysgdisk -Z /dev/XXXX will destroy GPT structures on that disk15:37
PavoI don't have sgdisk in CentOS15:37
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible: Refactor bootstrap.yml file  https://review.openstack.org/41550415:38
inc0erhudy, so immediate fix to your problem would be using template override mechanism15:38
*** jrich523 has quit IRC15:40
inc0http://docs.openstack.org/developer/kolla/image-building.html#dockerfile-customisation15:40
erhudyPavo: try gdisk package, it should come wit that15:40
Pavotrying to install it now15:40
Pavowouldn't fdisk do the same thing though?15:41
erhudyfdisk probably only understands MBR15:41
*** sacharya has joined #openstack-kolla15:41
erhudyinc0: there presently is not an override block in the base image dockerfile for what i care about, are you suggesting that your preference is that i rework the PR to add blocks for any sections where they don't already exist15:42
erhudythere are still some places where kolla itself would need modified in order to copy additional things into the build context15:42
inc0you can use base_apt_packages block15:42
inc0http://git.openstack.org/cgit/openstack/kolla/tree/docker/base/Dockerfile.j2#n25915:42
erhudyone other part of the PR re-works how apt keys are set up so that i can just give a directory of keys and then they're all copied into the context, instead of trying to reach an external keyserver15:42
inc0sorry, block name is base_ubuntu_package_installation15:43
erhudyokay, i wasn't sure if i had added that block or if it was upstream in master15:43
inc0you can extend this, write your code, modify sources.list/keys and do {{ super }}15:43
inc0but adding block in sources list is not bad as well15:43
inc0we can do as many blocks as it's reasonable15:44
inc0but regardless, http://docs.openstack.org/developer/kolla/image-building.html#ubuntu <- this should work15:44
erhudyalso the base_ubuntu_package_installation doesn't cover the section above where it still tries to use canonical's mirrors to add a few necessary packages15:44
inc0if it isn't it's a bug15:44
inc0I'll look into it15:44
erhudyspecifically the "apt-get -y install --no-install-recommends apt-transport-https ca-certificates" part15:44
erhudyso, yeah, you can see how this quickly turns into some yakshaving15:45
inc0base_header will do, you'd need to override both15:45
Pavoinc0 why would I get https://www.dropbox.com/s/62twu7a8nwxc4so/Screenshot%202017-01-12%2010.44.59.png?dl=0 only thing I have changed is now using btrfs instead of lvm15:45
inc0also good point, I'm not sure why we modify sources list *after* installation of ca-certs and stuff15:46
dmsimardinc0: oh hai, been meaning to talk to you -- I wanted to know what was the status on the openstack-infra/upstream registry15:46
*** yingjun has quit IRC15:46
inc0dmsimard, we don't have registry per se15:46
erhudyi think it was done that way initially because canonical's bare image lacks a few pieces that are then required for accessing https-only repos15:46
inc0Pavo, check logs man:)15:46
openstackgerritcaoyuan proposed openstack/kolla-ansible: Move octavia precheck into its own role  https://review.openstack.org/41953115:46
dmsimardinc0: so raw image hosting then?15:47
inc0erhudy, yeah, but that would mean someone would need to provide it over http...15:47
erhudyyeah, which canonical does, their repos can be accessed via either15:47
inc0dmsimard, it's a bit more hacky, hold on15:47
inc0dmsimard, http://tarballs.openstack.org/kolla/images/15:47
inc0this is literally tar.gz of registry15:47
inc0docker run registry:2 -v /tmp/somedir15:48
inc0then we tar.gz /tmp/somedir15:48
dmsimardinc0: interesting, that's what you settled for ? I remember you vigorously wanting a proper registry in infra :)15:48
inc0settled? no.15:48
inc0but that's something we can do to achieve immediate result of having gates working like they should15:49
*** jmccarthy has quit IRC15:49
inc0without infra changing anything15:49
erhudybasically my PR comes out of the fact that certain places in the bloomberg infra have absolutely no internet access, and assumptions of internet access are very pervasive and hard to stamp out15:49
inc0we still need to provide proper registry, but that would mean figuring few things out15:49
inc0mirroring, auth and such15:50
*** adrian_otto has joined #openstack-kolla15:50
inc0erhudy, that's not just you15:50
*** sp__ has joined #openstack-kolla15:50
inc0this isnt unusual use case15:50
dmsimardinc0: ok -- so how would I use one of those? I'm assuming I would download that, extract it and somehow point a private registry to that ?15:50
inc0but I think all the people who tried that uses centos so far15:50
inc0so ubuntu might be buggy on that front15:50
inc0when you run registry, run it with flag -v /var/lib/registry:/yourtempdir15:51
inc0dmsimard, https://github.com/openstack/kolla/blob/master/tools/setup_gate.sh#L15615:51
dmsimardinc0: okay, last question before I let you go :)15:51
*** adrian_otto has quit IRC15:51
dmsimardinc0: when are those generated ? on post merge ? on tags ?15:51
inc0post merge15:52
erhudyright, so, i guess my main point is that right now the ubuntu checks rely on a particular bug in kolla to work, and since my PR is failing on that it's impeding discussion about the approach because everyone sees "oh it failed jenkins, dev broke something"15:52
inc0this is master15:52
*** mliima has joined #openstack-kolla15:52
*** adrian_otto has joined #openstack-kolla15:52
erhudyi can sever that bugfix from the rest of the PR and present it separately so that it's not coming as part of a big package of change15:52
inc0yeah, thanks for bringing it up15:52
*** mliima_ has quit IRC15:53
erhudybecause, yeah, my PR does kind of change a lot of crap, so if it helps to reduce confusion, i'll re-do just that sources.list piece as a separate review15:53
dmsimardinc0: ok, ty15:53
erhudyinc0: thanks for responding on the issue, in any event15:56
openstackgerritMerged openstack/kolla: Heat configuration param change for heat-api section  https://review.openstack.org/41933015:58
*** pcaruana has joined #openstack-kolla15:58
*** absubram has joined #openstack-kolla15:59
openstackgerritcaoyuan proposed openstack/kolla-ansible: Move panko precheck into its own role  https://review.openstack.org/41953916:00
inc0erhudy, as I mentioned, afaik Pavo is doing same thing, but with centos (right Pavo? builidng without internet at all?)16:00
Pavoroger16:01
Pavoits not hard, just gotta add your repo files to /usr/share/kolla/docker/base directory and add rpm_setup_config = nameofcustomrepofile in kolla-build.conf16:02
erhudyyeah, but we're using ubuntu, not centos/RHEL, so16:02
Pavothat tells kolla-build to point to whatever repo you have internally16:02
erhudymaybe we're forging the path on the ubuntu stuff16:03
inc0yeah, you're first that I know of tbh16:03
erhudyRIP me then16:03
Pavohttp://docs.openstack.org/developer/kolla/image-building.html16:04
Pavounder custom repos16:04
inc0erhudy, well truth be told when we deal with this one problem16:04
inc0which we need in any case16:04
inc0I don't think there will be anything potentially wrong in the future16:04
erhudyto accommodate ubuntu/debian it needs to deal with both the sources.list bit and getting the keys in without contacting an external keyserver, unless we're willing to just install packages unauthenticated16:05
erhudywhich could be done by clobbering the apt preferences, but it's not my favorite solution16:06
inc0yeah, but it will get us there fastest with least fallout16:06
inc0apt.prefernces + sources list above RUN apt-get install ca-stuff16:06
erhudyyeah, the problem there is the sources.list would have to come in two pieces16:07
erhudybecause if you add the full override sources.list16:07
erhudyand then run apt-get update16:07
inc0ahh16:07
inc0it won't even update?16:07
erhudyit will probably detonate because of HTTPS-only repos, which it can't contact because it lacks the CA certs and the apt HTTPS transport16:07
erhudyright16:07
inc0crap16:07
erhudyblame canonical for not putting them in their bare image!16:08
inc0I do16:08
*** goldyfruit has joined #openstack-kolla16:08
inc0problem is, out of 2 solutions other than that none of them would really fix it16:09
inc0but, I know what would16:09
erhudyso, if you look at what i did to the dockerfile in my PR, that's why i took the approach with a couple of embedded ifs16:09
inc0so how about wrapping apt-get install ca-stuff into block16:09
inc0and then you can do it manually somehow16:09
inc0even from raw package over dpkg -i16:10
erhudyuh, let me look at the dockerfile in master to see what would need to be put into blocks16:10
inc0so adding new block is free16:10
inc0not changing anything16:10
inc0you can propose patch with as many blocks as you want as long as names are unique16:10
erhudyi guess i still have questions about how i could copy additional files into the build context, because that seems like it requires kolla code changes no matter what16:10
inc0yeah I don't think docker understands COPY from absolute path16:11
erhudyit doesn't16:11
erhudywhich is why i ended up adding another config option that says "here's a directory of files that will get fed to apt-key, since you can't contact an external keyserver"16:11
*** sacharya_ has joined #openstack-kolla16:11
inc0so you'd need to copy deb into kolla source tree16:11
inc0alternatively we can do it in our build script16:12
inc0have additional cp during build16:12
*** jrich523 has joined #openstack-kolla16:12
erhudymaybe we can compromise on my PR and leave some of the extra config options in that i added, but remove some of the convoluted embedded ifs and replace them with straightforward blocks that we can override16:13
*** sayantani01 has joined #openstack-kolla16:13
inc0yeah, blocks would be much cleaner16:13
inc0we still need to solve this issue tho16:13
erhudythen there's not all sorts of logic being embedded in the template, but we can still add the necessary files to the build context16:13
erhudythis issue being the bug about not reading sources.list at all, or something else?16:13
inc0it does read sources.list, has to - we do use mirrors in gates16:14
*** sacharya has quit IRC16:14
erhudyin my e-mail i linked a log file from another build that rebuilt base16:14
erhudyit goes straight to canonical, never touches OSIC16:14
*** rbergeron has quit IRC16:14
Pavoinc0 this is what horizon log says http://paste.openstack.org/show/594729/16:14
*** jmccarthy has joined #openstack-kolla16:15
Pavoany idea16:15
inc0erhudy, https://github.com/openstack/kolla/blob/2c887565d4299462a29b98210c2a7ce233e71039/kolla/image/build.py#L58716:15
erhudyinc0: i know, it copies sources.list into the context16:15
erhudybut16:15
inc0Pavo, check keystone log16:15
erhudyhttps://github.com/openstack/kolla/blob/2c887565d4299462a29b98210c2a7ce233e71039/docker/base/Dockerfile.j2#L21416:15
erhudyit doesn't ever get used16:16
inc0no, hold on16:16
inc0our build script does something else16:16
erhudyit will always copy sources.list.ubuntu or sources.list.debian to /etc/apt/sources.list16:16
inc0ahh16:16
inc0yeah, good point16:16
inc0so THAT is a bug16:16
inc0we need to copy sources.list with name suffix16:16
erhudyso, yeah, when i realized that and fixed that, now the sources.list in the gate _does_ get used, and it's not signed, and so everything falls apart16:16
inc0fun life16:17
inc0so we should remove gate mirrors for now16:17
inc0can you do it in your patchset?16:17
inc0I'll comment why16:17
inc0(or you can;))16:17
erhudycan you do it so that it doesn't just look like i'm shouting into the abyss16:17
inc0it's funny because literally we announced debian deprecation today16:18
openstackgerritJeffrey Zhang proposed openstack/kolla: Use ZUUL_REFNAME to get branch info during pack registry  https://review.openstack.org/41954816:18
inc0ok16:18
Pavoinc0 not seeing any errors in keystone.log16:18
inc0mind if I do it in your patchset?16:18
erhudyoh, if we ignore debian, that's great, idc about debian16:18
erhudynot at all16:18
inc0Jeffrey4l, ^16:18
inc0did you know that our ubuntu gates never used mirrors?16:18
Pavoinc0 only getting Warning of deprication16:19
Jeffrey4linc0, yep.16:19
Jeffrey4lwait.16:19
*** sayantani01 has quit IRC16:20
*** dag_ has joined #openstack-kolla16:20
Jeffrey4linc0, , centos doesn't and ubuntu does.16:20
*** dag_ is now known as rbergeron16:21
Jeffrey4lcould u review this https://review.openstack.org/#/c/419548/ , we are using a wrong variable when packaging registry.16:21
Jeffrey4linc0, ^16:21
inc0Jeffrey4l, no, what I mean, custom apt soruces.list doesn't work at all16:21
Jeffrey4lfile name is wrong here http://tarballs.openstack.org/kolla/images/16:21
*** rbergeron has quit IRC16:21
*** rbergeron has joined #openstack-kolla16:21
Jeffrey4lmaybe. i saw some fix in queue. but i never used ubuntu ;/16:22
inc0yeah we were just discussing that16:22
erhudyi guess i'm learning that kolla is a centos world and i'm the lone crazy person using ubuntu :v16:22
sdake_morning folks16:22
inc0erhudy, I'm ubuntu too16:22
sdake_looks like my calendar was off - tomorrow is dentisst torture day16:22
*** salv-orl_ has joined #openstack-kolla16:23
Jeffrey4linc0, i think we can move to dockerfile customization solution for this.16:23
openstackgerritEduardo Gonzalez proposed openstack/kolla: Add missing pull task for neutron-vpnaas-agent  https://review.openstack.org/41955016:24
Jeffrey4lno need use apt_sources-list and apt_preferences parameter16:24
inc0Jeffrey4l, not exactly unfortunately, docker can't COPY absolute path16:24
Jeffrey4lwe can use something trick, base64 ;)16:25
erhudynoooooooooooooo16:25
*** salv-orlando has quit IRC16:26
Jeffrey4lerhudy, inc0 check here, not that bad https://github.com/openstack/kolla/blob/master/tools/setup_gate.sh#L3716:26
sdake_Jeffrey4l did we have any conclusion with sbezverk on the gate problem with the 3.0.2 images?16:27
Jeffrey4lsdake_, no. may need kfox1111 to check the script.16:27
Jeffrey4lgate scripts.16:27
Jeffrey4li am not familiar with that.16:28
sdake_Jeffrey4l roger16:28
sdake_Jeffrey4l so you believe the gate scripts are incorrect then?16:28
sdake_Jeffrey4l one q i'd have about that is why did it work with 3.0.1 images?16:28
Jeffrey4lsdake_, either gate script is wrong or k8s use some ceph command in wrong.16:29
sdake_Jeffrey4l ok thanks dude for the hep with the triage16:29
Jeffrey4li guess ceph package is updated?16:29
bjolokeystone and memcached16:29
sdake_Jeffrey4l it is possible we went from 0.94.0 to 0.94.116:29
sdake_I am uncertain on that point16:29
bjoloany reason why kolla newton is not using memcached for tokens by default_16:30
sdake_although librdb may have changed as well16:30
bjolo?16:30
sdake_bjolo iirc it had something to do with security16:30
sdake_bjolo SamYaple might know more about it16:30
bjolook tnx16:30
Jeffrey4l3.0.2 is using ceph 10.2.216:30
sdake_huh?16:30
sdake_what is 3.0.1 using?16:30
sdake_let me connect and check myself16:31
sdake_brb16:31
Jeffrey4lpulling the image.16:31
*** adrian_otto1 has joined #openstack-kolla16:32
*** adrian_otto has quit IRC16:32
*** sdake has joined #openstack-kolla16:33
jemcevoyMorning Everyone,  I am trying to figure a way to get a smooth install to work using the vlan provider16:33
jemcevoythat Mech422 shared with me yesterday http://www.opencloudblog.com/?p=614 ... I see a chicken and16:33
jemcevoyegg issue with OVS. I will need a OVS bridge setup for the kolla config but the kolla install will16:33
jemcevoywipe the ovs database when the openvswitch_db get loaded.   Is there a way to install the openvswitch containers in my prep palybooks?16:33
erhudyinc0 Jeffrey4l: i will go back to my PR and look at where to add additional blocks to accommodate what we want to do16:33
erhudyi can't say base64-encoding apt keys into a block override is my favorite solution of all time, but, hacks make the world go around16:34
sdakeerhudy ya - its imperct - although i htink best we have come up with atm :)16:34
Mech422jemcevoy: I just hacked the kolla container not to restart openvswitch16:34
sdakeerhudy that is also how the gate works (or used to work)16:35
*** sdake_ has quit IRC16:35
*** zhubingbing has quit IRC16:36
sdakeJeffrey4l 3.0.1: ceph-10.2.2-0.el7.x86_6416:36
sdake3.0.2: ceph-10.2.2-0.el7.x86_6416:36
Jeffrey4l3.0.2   ceph-common-10.2.2-0.el7.x86_64  ok it is the same.16:36
sdake(pulled from hub)16:36
Jeffrey4lthen no idea :(16:37
sdakethere is a rdb library16:37
sdaketryign to find the package that is in16:37
sdakemaybe that has  changed16:37
sdake3.0.2: librbd1-10.2.2-0.el7.x86_6416:37
sdake3.0.1: librbd1-10.2.2-0.el7.x86_6416:38
sdaketheoretically - nothing has changed :)16:38
sdakekfox1111 are you around?16:38
sdakeor sbezverk around?16:38
Jeffrey4lyep. weird16:38
sdakecould be a kernel changef issue16:38
sdakei'm not sure which patchset produced that librdb error16:39
openstackgerritJeffrey Zhang proposed openstack/kolla: TEST NEWTON BRANCH  https://review.openstack.org/41703316:40
sbezverksdake: hey what is up?16:41
jemcevoyMech422, So where does your  openvswitch data base end up in the container or on the bare metal?   I will access to a VLAN on the host for the hypervisors to work with ceph16:41
Mech422mine was on the metal16:42
kfox1111morning.16:42
openstackgerritJeffrey Zhang proposed openstack/kolla: Fix ceilometer idempotent  https://review.openstack.org/38631416:43
sbezverksdake: if it is about ceph issue I found this bug: http://tracker.ceph.com/issues/1628916:43
sbezverkkfox1111: I might hit global api issue with service level packages16:43
kfox1111sbezverk: should work. I was able to munge the neutron stuff in I think: https://review.openstack.org/#/c/412239/16:44
kfox1111made for a much easier helm_entrypoint_general: https://review.openstack.org/#/c/412239/22/tests/bin/setup_helm_entrypint_config.sh16:45
kfox1111sdake: I'm going to try reverting to 2.0.2 for everything but the previous 3.0.1 supporting stuff.16:45
kfox1111its been a very long time since we tried 3.0.1 for everything I think.16:45
sbezverkkfox1111: the issue is with all the same settings , glance-api does not get logger container but glance-registry does..16:46
kfox1111maybe we never had ceph in the gate working with 3.0.1. don't remember. its been too long.16:46
kfox1111sbezverk: weird. that seems like a bug...16:46
sdakekfox1111 sound sgood so some images (like toolbox) will be 3.0.2?16:46
sbezverkkfox1111: http://logs.openstack.org/36/410736/29/experimental/gate-kolla-kubernetes-deploy-centos-binary-helm-entrypoint-nv/34bfc23/console.html16:46
kfox1111a config like https://review.openstack.org/#/c/412239/22/tests/bin/setup_helm_entrypint_config.sh should work for that.16:46
sbezverkwhen you have a chance to check..16:47
kfox1111sdake: yeah. I'll leave those at 3.0.2 as it fixes bugs.16:47
sbezverkbb in 30 minutes16:47
kfox1111sbezverk: ok. give me a few to try and fix up the gate.16:47
kfox1111k16:47
sdakekfox1111 sounds good16:47
openstackgerritsayantani proposed openstack/kolla: Improve Quick Start documentation  https://review.openstack.org/40499316:47
sdakekfox1111 btw i did manage to push ubuntu and centos last night, although ubuntu was built on centos host16:47
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: Set versions to 0.4.0 and 3.0.2  https://review.openstack.org/41908216:48
kfox1111sdake: cool. thanks.16:48
sdakekfox1111 so inc0 may want to push the proper 3.0.2 images for ubuntu as I am out of disk space on my laptop where I run vms16:48
kfox1111we should know in about 20 min.16:48
kfox1111k.16:48
sdakekfox1111 sweet - then we are good to tag 0.40?16:48
inc0I think they're there16:48
kfox1111if you don't mind having a 2.0.2 based 0.4.0.16:49
kfox1111works well though.16:49
inc0https://hub.docker.com/r/kolla/ubuntu-binary-kolla-toolbox/tags/ <- 3.0.2 is there16:49
*** jmccarthy has quit IRC16:50
kfox1111heh. their dates are off: https://hub.docker.com/r/kolla/ubuntu-binary-kubernetes-entrypoint/tags/16:50
kfox1111there wasn't a 3.0.2 until yesterday.16:50
sdakeinc0 i uploaded that last night16:51
sdakeinc0 if you uploaded as well, i may have ovewritten yours16:51
sdakeinc0 so you may need to do a push again if its not too painful16:51
sdakebtw- who is uploading master images to dockerhub16:52
sdakeanyone know?16:52
sdakeit almost looks like a gate job16:52
sdakemaster tags i mean16:52
*** sayantani01 has joined #openstack-kolla16:53
sdakeeveryone: there is a lag between docker push and the hub tag metadata being updated in the web ui16:53
sdakeit seems about a  1-2 hour lag16:53
sdakeinc0 so I thought you didn't upload (if you did) and uploaded as it was blocking kfox111116:53
sdakeinc0 hope that adds a bit of clarity16:54
Jeffrey4lwhy kolla-k8s do not build images in gate ;/16:54
kfox1111sdake: portdirect is uploading master images. :)16:54
kfox1111he's got a cron job going, so it will keep them up to date. :)16:55
sdakekfox1111 ok - we weren't doing that in the past because they are test images - however seems fine to me16:55
kfox1111yeah. I think they are just intended for testing. but still useful.16:55
sdakekfox1111 medium term once the fires are put out we should be using master images from hub rather then 3.0.z16:55
Jeffrey4lit is  super easy now to build the image you want.16:55
sdakeJeffrey4l everyoen wants the stuff on docker hub for convience16:56
Jeffrey4lbut for master branch gate testing. it is not convience to depends on hub.docker.com16:56
portdirectJeffrey4l: I try to develop on my own workstation rather than just reply on the gate - for that it is easier to pull, than have to build each time16:57
portdirect*rely16:57
openstackgerritsayantani proposed openstack/kolla: Improve Quick Start documentation  https://review.openstack.org/40499316:58
Jeffrey4li just wanna to talk the gate. kolla-ansible build the image in gate now. rather than use a hub.docker.com edition. and the issue is there is no master branch image too in docker.16:59
*** fooliouno has joined #openstack-kolla17:00
*** sdake_ has joined #openstack-kolla17:02
sayantani01Hi guys, as discussed, the documentation team wants to add the quickstart guide to the installation guides section of openstack. It will be great if I can have some eyes on https://review.openstack.org/#/c/404993/ .17:02
*** sdake has quit IRC17:04
openstackgerritDan Prince proposed openstack/kolla: Add Apache packages to cinder-api container  https://review.openstack.org/41907617:05
*** mliima_ has joined #openstack-kolla17:06
klindgrenHey all - why was this change abandoned? https://review.openstack.org/#/c/412231/  I didn't sense any descent in the reviews - so I am more than willing to rebase/take it over.  But it seemed like it was at a pretty good state.17:06
*** unicell has quit IRC17:07
*** wcb has joined #openstack-kolla17:07
openstackgerritDan Prince proposed openstack/kolla: Add Apache packages to nova-api container  https://review.openstack.org/41813117:07
*** mliima has quit IRC17:07
*** williamcaban has quit IRC17:07
*** afranc has quit IRC17:07
*** afranc has joined #openstack-kolla17:10
sayantani01Guys, this is where the kolla deployment guide will be after the doc team takes over http://docs.openstack.org/project-deploy-guide/newton/17:10
sdake_klindgren let me unabandon it17:10
sdake_moment17:10
sdake_klindgren not sure why that change was abandoned - the last statement from SamYaple was that someone could take it over17:12
sdake_klindgren all yorus :)17:12
*** fallingsky has joined #openstack-kolla17:12
klindgrenkk17:12
sdake_klindgren note Jeffrey4l I think was doing some work in this area - might sync with him17:12
sdake_Jeffrey4l or SamYaple may be able to add more context a to why it was abandoned17:13
Jeffrey4lkfox1111, it is not abandoned. just sam -w to wait for feedback.17:13
Jeffrey4lsorry, ping the wrong person.  sdake_ klindgren ^^17:14
sdake_Jeffrey4l it was abaonded - i just restored it17:14
Jeffrey4lok got it. sam abandonded it.17:14
sdake_Jeffrey4l sam indicated in the comments section that anyone else was free to take it over17:14
Jeffrey4lwe should continue that works.17:14
*** fooliouno has quit IRC17:14
sdake_Jeffrey4l cool - looks like a green light to me klindgren  :)17:15
Jeffrey4lno idea why Sam did that ;(17:15
*** sayantan_ has joined #openstack-kolla17:15
*** bjolo has quit IRC17:16
klindgrenkk - I have no problem shepherding this.17:17
*** sayantani01 has quit IRC17:19
*** bmace has quit IRC17:19
*** bmace has joined #openstack-kolla17:20
sdake_sayantan_ looks great - can't wait for that work to finish :)17:20
sayantan_sdake_ Thank you. Couldn't agree more :P17:20
*** portdirect is now known as portdirect_eatin17:21
Pavowow firewalld was stopping from Bootstrap Ceph OSD from working17:22
Pavo*facepalm*17:22
inc0sdake_, can I have your eyes on this one? https://review.openstack.org/#/c/404993/17:24
inc0to help sayantan_ with deploy guide17:24
inc0we wanted it to be kinda similar/exactly the same17:24
sdake_inc0 yup i actually have it open now17:24
*** msimonin has quit IRC17:24
sayantan_inc0 , sdake_ Thanks :)17:25
*** matrohon has quit IRC17:26
*** jmccarthy has joined #openstack-kolla17:26
kfox1111hmm... the hub seems to be still a bit flaky.17:26
kfox1111sdake_: but the 2.0.2 ceph does seem happy.17:27
mliima_just fix merge conflict and it'll be ready to go https://review.openstack.org/#/c/412231/ klindgren17:27
klindgrenyep - working on that now17:27
sdake_kfox1111 using 3.0.1 images?17:27
kfox1111using 2.0.2 images except where 3.0.2 is required.17:27
sdake_kfox1111 I looked at centos binary - and it looks to me like its the same exact ceph release from the virtualization sig17:27
sdake_kfox1111 oh - your jumping from 2.0.2 to 3.0.2 ?17:28
kfox1111sdake_: yeah. I don't know if we ever tested the ceph deployment parts of the gate with 3.0.x.17:28
sdake_3.0.2 uses a completely different version of ceph (jewel) then 2.0.2 (hammer)17:28
kfox1111so its possible there is an issue with ceph in 3.0.x package.17:28
sdake_kfox1111 Jeffrey4l tested ceph multinode ansible and that works17:28
kfox1111yeah.17:28
kfox1111its just that one weird issue I think.17:29
inc0yeah I have jewel deployed even now17:29
kfox1111where it won't take the image creation request.17:29
sdake_my guess is from hammer to jewel, the CLI changed in some odd way17:29
inc0ceph changed way it uses permissions between versions17:29
sdake_inc0++ :)17:29
sdake_kfox1111 the permissions change would do it17:29
sdake_had forgotten about that17:29
sdake_kfox1111 also your probably running with selinux enabled17:30
sdake_kfox1111 which we do in the gate jobs17:30
kfox1111no. selinux is disabled17:30
kfox1111if we can figure out:17:30
kfox11112017-01-12 02:21:10.080105 | + kubectl exec ceph-admin -c main --namespace=kolla -- /bin/bash -c 'timeout 240s rbd create kollavolumes/mariadb --size 1024'17:30
kfox11112017-01-12 02:21:11.929763 | 2017-01-12 02:21:11.928333 7f40cc4d3d80 -1 librbd: error writing header: (38) Function not implemented17:30
*** jmccarthy has quit IRC17:30
kfox11112017-01-12 02:21:11.933492 | rbd: create error: (38) Function not implemented17:30
sdake_kfox1111 oen thing to try is to disable selinux - although thats not a long term solution17:30
sdake_inc0 what do you make of that above?17:30
inc0kfox1111, you're using 2.0.2 across the board?17:30
kfox1111inc0: mostly.17:31
sdake_kfox1111 i did see that yesterday when you posted teh full log - but not sure if inc0 has seen it or SamYaple has seen it, and they are both experts in ceph17:31
inc0looks like librbd version conflict17:31
kfox1111all the other ceph stuff seems to be working.17:31
*** egonzalez has left #openstack-kolla17:31
kfox1111shouldn't be. everything should be 3.0.2 in that gate.17:31
inc0isn't librbd also kernel module?17:32
inc0it changed too I think17:32
inc0at some point17:32
sbezverkkfox1111: have you had a chance to check the log I pasted?17:33
kfox1111the kernel module hasn't changed in a long time.17:33
kfox1111sbezverk: not yet. sorry. trying to figure out why 3.0.2 isn't working.17:33
sbezverkkfox1111: here is the bug with exact error we are getting: http://tracker.ceph.com/issues/1628917:33
inc0what I'm saying is they used kernel module but now they push librbd lib instead17:33
sbezverkI suspect you have done some magic with ceph, and somewhere some binaries version maybe got hardcoded..17:34
kfox1111sbezverk: yeah, that looks about right. but I'm not sure how to parse it.17:34
inc0I'm fairly sure glance is supposed to use image layering17:34
kfox1111inc0: this code is creating volumes for k8s, not for glance.17:35
kfox1111its possible we need some flags maybe for jewel that weren't required for hammer?17:36
sbezverkkfox1111: the thing is I use jewel in my cluster and it is wroking fine with kube17:36
sbezverkthe only diff is I am not using ceph containers you introduced17:37
inc0yeah...maybe you could use kolla containers for ceph?;)17:37
kfox1111you did the same command I did? rbd create kollavolumes/mariadb --size 102417:37
sbezverkrbd create --pool volumes --image-feature layering --size 10240 mariadb17:39
inc0rbd create rbd/mariadb --size 1024 <- works just fine17:39
inc0works with layering too17:39
sbezverkinc0: this is from local cluster17:39
sbezverkit was/is and will be working ;-)17:40
inc0ubuntu@inc0-kubernetes-1:~$ rbd --version17:40
inc0ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374)17:40
kfox1111so, maybe --image-feature layering is requried now?17:40
sbezverkkfox1111: I think so17:40
inc0kfox1111, I tried both with and without17:40
inc0works just fine both ways17:40
sbezverkI remember reading it somewhere now that you mentioend it17:40
*** mgiles has quit IRC17:40
inc0it does look like client error tho17:40
sbezverkrbd --version17:41
sbezverkceph version 10.2.517:41
inc0ubuntu@inc0-kubernetes-1:~$ dpkg -l | grep librbd17:41
inc0ii  librbd1                              10.2.2-0ubuntu0.16.04.2             amd64        RADOS block device client library17:41
inc0ii  python-rbd                           10.2.2-0ubuntu0.16.04.2             amd64        Python libraries for the Ceph librbd library17:41
kfox1111easy enough to test.. lets try it.17:42
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: Set versions to 0.4.0 and 3.0.2  https://review.openstack.org/41908217:42
kfox1111sbezverk: I dug into the ps attached to the issue you posted. it looks like it was about adding the option to specify extra flags. so I think that does imply it needs an extra flag.17:44
kfox1111brb17:45
sdake_sayantan_ - NICE WORK17:52
sdake_sayantan_ first time the quickstart has been rewritten (many people have tried) without totally breaking how it works ;)17:53
kfox1111back17:53
sayantan_Thanks sdake_ Will address your comments now :)17:53
sbezverkkfox1111: have you seen it? http://paste.openstack.org/show/594749/17:56
inc0sayantan_, I made few comments too, but it's almost ready17:58
sayantan_inc0 Thanks, working on all the comments now. :)17:58
kfox1111sbezverk: usuauly when there's a left over directory from switching branches. :/17:59
kfox1111see if /helm/service/neutron looks incomplete. if it is, can just rm -rf it.18:00
kfox1111if we really wanted to fix that, we'd have to put a list of packages into the prebuild scripts.18:00
kfox1111rather then just doing all found dirs.18:00
kfox1111as switching git branches can sometimes leave some dirs around.18:01
*** msimonin has joined #openstack-kolla18:01
sbezverkkfox1111: let me manually clean them up, hopefully it will help..18:01
*** Serlex has quit IRC18:03
kfox1111hmm... the layering flag let it get much farther.18:04
kfox1111http://logs.openstack.org/82/419082/6/check/gate-kolla-kubernetes-deploy-centos-binary-external-ovs-nv/c28da21/console.html18:04
sbezverk:-)18:04
kfox1111http://logs.openstack.org/82/419082/6/check/gate-kolla-kubernetes-deploy-centos-binary-external-ovs-nv/c28da21/logs/openstack/kolla-glance-api-251924186-0r8z3.txt18:05
kfox1111glance error creating image...18:05
*** unicell has joined #openstack-kolla18:07
*** msimonin has quit IRC18:07
sbezverkkfox1111: that would mean older ceph backend binaries18:07
kfox1111it all should be 3.0.2 though.18:08
sbezverkkfox1111: we should be able to see it in pod definition if it uses 3.0.2 image18:10
kfox1111yeah. let me look.18:11
kfox1111ah... I forgot not all of the ceph stuff is in microservice packages yet. its still using kollakube for it. gotta tweak versions there too.18:13
*** msimonin has joined #openstack-kolla18:14
*** srwilkers is now known as srwilkers_aaway18:14
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: Set versions to 0.4.0 and 3.0.2  https://review.openstack.org/41908218:14
*** sacharya has joined #openstack-kolla18:17
*** sacharya_ has quit IRC18:17
*** sp__ has quit IRC18:17
sbezverkkfox1111: once ceph gets sorted out, appreciate if we could take a look at global api.. as glance-api and glance-registry use identical variables, at least for general things, glance-api correcty see that no logger and hapoxy is needed, but glance-registry keep adding these containers. The main difference is glance-api uses its own template but glance-registry uses common api python18:19
sbezverktemplate..18:19
kfox1111K. I'm waiting on zuul now, so I'll have a look now.18:20
*** msimonin has quit IRC18:21
*** msimonin has joined #openstack-kolla18:21
*** msimonin has quit IRC18:21
*** msimonin has joined #openstack-kolla18:22
*** msimonin has quit IRC18:22
*** msimonin has joined #openstack-kolla18:22
*** msimonin has quit IRC18:23
*** msimonin has joined #openstack-kolla18:23
*** msimonin has quit IRC18:24
*** msimonin has joined #openstack-kolla18:24
*** msimonin has quit IRC18:25
*** msimonin has joined #openstack-kolla18:25
*** msimonin has quit IRC18:25
*** jmccarthy has joined #openstack-kolla18:29
openstackgerritKris Lindgren proposed openstack/kolla: use static uid/gid in images  https://review.openstack.org/41223118:31
*** portdirect_eatin is now known as portdirect18:32
kfox1111sbezverk: are you working on: https://review.openstack.org/#/c/410736 ?18:32
sbezverkkfox1111: yep18:32
*** openstackgerrit has quit IRC18:33
kfox1111I'm still debugging, but tyr removing the helm_entyrpoint_glance section from https://review.openstack.org/#/c/410736/29/tests/bin/ceph_workflow_service.sh, and pulling in this: https://review.openstack.org/#/c/412239/22/tests/bin/setup_helm_entrypint_config.sh18:34
*** pcaruana has quit IRC18:34
kfox1111and you can add the ceph bits in there too.18:34
kfox1111shouldn't have to do individual package config's anymore most of the time with the global api.18:35
kfox1111as for the kubernetes_entrypoint vars, I'd stick them in the service packages values.yaml. like:18:36
kfox1111https://review.openstack.org/#/c/412239/22/helm/service/neutron/values.yaml18:36
kfox1111so they get set automatically when you use that package.18:36
*** prameswar has quit IRC18:36
sbezverkkfox1111: ok18:36
*** jmccarthy has quit IRC18:37
kfox1111ok... looks like I got rid of all of the 2.0.2 containers. and it got further.18:39
kfox1111its complaining about lack of flavors now.18:40
kfox1111does newton not prebuild flavors anymore?18:40
sbezverkkfox1111: yeah, it comes clean18:40
*** ipsecguy has quit IRC18:41
*** mannamne has quit IRC18:42
kfox1111ok. probably just need to add the mini flavor.18:43
satyarHi jeffrey4l18:45
*** openstackgerrit has joined #openstack-kolla18:45
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: Set versions to 0.4.0 and 3.0.2  https://review.openstack.org/41908218:45
satyarregarding https://review.openstack.org/#/c/404944/22/ansible/library/rabbitmq_hostname.py@67 i am not suspecting we will get the error18:46
sbezverkkfox1111: ceph related parameters still needs to be added at the run time, since monitor ip changes for multinode case18:46
satyaras we will be getting the IPs from ansible facts18:46
satyarjeffrey4l18:46
sbezverkkfox1111: I mean for glance pv18:46
*** fguillot has joined #openstack-kolla18:47
kfox1111sbezverk: yeah. I think thats covered in:18:47
kfox1111https://review.openstack.org/#/c/412239/22/tests/bin/setup_helm_entrypint_config.sh18:47
*** vinkman has joined #openstack-kolla18:47
kfox1111can just add the appropriate variables to the list?18:47
kfox1111like, I think external_vip may need to change for multinode. don't quite remember.18:48
kfox1111can set the ceph bits in there too.18:48
*** cliles has joined #openstack-kolla18:49
sbezverkkfox1111:  ok will try it18:49
*** rhallisey has quit IRC18:52
*** MasterOfBugs has joined #openstack-kolla18:52
*** PramodJ has joined #openstack-kolla18:52
*** rhallisey has joined #openstack-kolla18:54
openstackgerritsayantani proposed openstack/kolla: Improve Quick Start documentation  https://review.openstack.org/40499318:56
openstackgerritEdmund Rhudy proposed openstack/kolla: Add blocks for Ubuntu apt setup and package installation  https://review.openstack.org/41961019:00
erhudylet's try that as a slightly lower-touch version of the last PR19:03
sayantan_kfox1111 inc0 sdake_ srwilkers_ portdirect I have addressed all your comments on the quickstart guide :)19:05
kfox1111sayantan_: cool. thanks. :)19:05
sdake_sayantan_ i hav ea dr appointment i need to head out for in about 5 mins so won't have time to do a review19:05
sdake_sayantan_ I will review once I return from that + picking up daughter from school19:05
sayantan_It will be great if you can review it again so that we can move forward19:06
sayantan_Sure sdake_ no rush19:06
sdake_sayantan_ agreed I will do that - in 1-2 hours - my word19:06
*** sdake_ is now known as sdake19:06
sayantan_Thanks sdake_ Appreciate it :)19:06
sdakenp19:07
*** vinkman has left #openstack-kolla19:07
Pavohttps://www.dropbox.com/s/m13annrmbmpja8f/Screenshot%202017-01-12%2014.12.34.png?dl=019:12
Pavonot bad at all19:12
kfox1111weird...19:17
inc0it is not bad, 144 cores per server19:17
inc0I've seen worse;)19:17
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: Set versions to 0.4.0 and 3.0.2  https://review.openstack.org/41908219:17
kfox1111http://logs.openstack.org/82/419082/8/check/gate-kolla-kubernetes-deploy-centos-binary-external-ovs-nv/f4afbae/console.html private shoudl have worked, as I didn't change tenants...19:18
inc0erhudy, you don't technically need to wrap packages into block19:18
satyarHi inc019:18
kfox1111I wonder if thats another memcached issue. :/19:18
inc0stuff {% set=[] %} can be modified in a different way19:18
*** shardy has quit IRC19:18
erhudyokay, i'll revise it once the gates are done with it, in case something else needs fixed19:19
inc0reviewed19:20
inc0one more comment besides this one19:20
erhudybackwards compatibility in what sense? adding another layer?19:22
inc0no, if someone has override file for that block19:22
inc0that will modify key installation19:22
erhudyah19:22
inc0it will stop working or work in a different way19:22
inc0not sure which one is worse;)19:22
inc0I'm not sure if you can't do nested blocks19:23
inc0I think you can19:23
inc0that would solve it19:23
erhudyafaik jinja will allow it19:24
inc0check it out, then you can create 2 nested blocks the way you did and wrap it above19:25
inc0or, if you don't actually need this granularity19:25
inc0just leave it as it is, your call19:25
*** Pavo has quit IRC19:26
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes: Glance service orchestration with entrypoint  https://review.openstack.org/41073619:26
openstackgerritEdmund Rhudy proposed openstack/kolla: Add blocks for Ubuntu apt setup and package installation  https://review.openstack.org/41961019:27
*** jmontleon has joined #openstack-kolla19:27
sbezverkkfox1111: we do not checkin values.yaml, how do you push your neutron's values.yaml so it is get used at the gate?19:28
kfox1111git add -f helm/servicepackage/...19:29
kfox1111I always do two step commits.19:29
kfox1111git add19:29
kfox1111git commit19:29
kfox1111that way I know whats goign into the commit.19:29
kfox1111but I'm paranoid. :)19:29
*** rhallisey has quit IRC19:30
inc0erhudy, looks good19:30
*** rhallisey has joined #openstack-kolla19:30
inc0I'll fix the gates in the meantime19:30
erhudyradical19:30
kfox1111sbezverk: commented on the review.19:31
*** srwilkers_aaway is now known as srwilkers19:36
srwilkersafternoon o/19:36
kfox1111srwilkers: hey19:36
*** satyar has quit IRC19:37
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes: Glance service orchestration with entrypoint  https://review.openstack.org/41073619:43
sbezverkkfox1111: thanks, addressed them now keeping fingers crossed cannot type ;-)19:44
kfox1111yeah. that looks better. should have a fair chance of working now I think.19:45
*** mgoddard has joined #openstack-kolla19:46
openstackgerritMerged openstack/kolla: Fix ceilometer idempotent  https://review.openstack.org/38631419:46
kfox1111closer still.... http://logs.openstack.org/82/419082/9/check/gate-kolla-kubernetes-deploy-centos-binary-external-ovs-nv/366e0ed/console.html19:49
kfox1111got a meeting to go to. biab.19:50
*** tonanhngo has joined #openstack-kolla19:54
*** salv-orl_ has quit IRC19:54
*** mgoddard_ has joined #openstack-kolla19:56
*** mgoddard has quit IRC19:57
*** mgoddard has joined #openstack-kolla19:58
*** tonanhngo has quit IRC19:59
*** mgoddard_ has quit IRC20:01
*** salv-orlando has joined #openstack-kolla20:01
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes: WIP memcached operator  https://review.openstack.org/41807320:03
*** ipsecguy has joined #openstack-kolla20:06
-openstackstatus- NOTICE: Gerrit will be offline between now and 20:30 for scheduled maintenance: http://lists.openstack.org/pipermail/openstack-dev/2017-January/109910.html20:08
*** ChanServ changes topic to "Gerrit will be offline between now and 20:30 for scheduled maintenance: http://lists.openstack.org/pipermail/openstack-dev/2017-January/109910.html"20:08
*** sayantan_ has quit IRC20:09
*** adrian_otto1 has quit IRC20:17
*** schwicht has quit IRC20:17
*** tonanhngo has joined #openstack-kolla20:17
*** aj_power has joined #openstack-kolla20:17
*** schwicht has joined #openstack-kolla20:18
*** awiddersheim has joined #openstack-kolla20:21
*** awidders_ has quit IRC20:21
*** awidders_ has joined #openstack-kolla20:23
*** awiddersheim has quit IRC20:23
*** awiddersheim has joined #openstack-kolla20:25
*** awidders_ has quit IRC20:25
*** sayantani01 has joined #openstack-kolla20:27
*** aj_power has quit IRC20:29
*** awiddersheim has quit IRC20:31
*** awiddersheim has joined #openstack-kolla20:32
*** mannamne has joined #openstack-kolla20:32
-openstackstatus- NOTICE: Updated: Gerrit will be offline until 20:45 for scheduled maintenance (running longer than anticipated): http://lists.openstack.org/pipermail/openstack-dev/2017-January/109910.html20:33
*** ChanServ changes topic to "Updated: Gerrit will be offline until 20:45 for scheduled maintenance (running longer than anticipated): http://lists.openstack.org/pipermail/openstack-dev/2017-January/109910.html"20:33
*** jrist has quit IRC20:33
*** goldyfruit has quit IRC20:42
*** jrist has joined #openstack-kolla20:47
*** ChanServ changes topic to "Have support questions? Please give ask.openstack.org a try. New to Kolla? Please read the documentation here: http://docs.openstack.org/developer/kolla/; Kolla IRC meetngs on Wednesdays @ 16:00 UTC - see agenda @ https://goo.gl/OXB0DL - IRC channel is *LOGGED* @ http://goo.gl/3mzZ7b"20:53
sdakesbezverk are we good to go with the release?20:53
*** jrobinson has joined #openstack-kolla20:54
*** david-lyle has quit IRC21:01
*** david-lyle has joined #openstack-kolla21:02
sdakeinc0 about?21:05
inc0about to go to lunch21:08
inc0wassup?21:08
*** fallingsky has quit IRC21:09
*** adrian_otto has joined #openstack-kolla21:10
sean-k-mooneyis there a kolla ptg etherpad by the way.21:15
inc0indeed there is21:15
inc0https://etherpad.openstack.org/p/kolla-ocata-ptg21:16
*** mgoddard has quit IRC21:16
sean-k-mooneyi see its not useing the same nameing convention as nova https://etherpad.openstack.org/p/kolla-ptg-pike21:16
sean-k-mooneyah ok cool thanks21:16
inc0yeah it's technically pike21:16
inc0well, next one will be R21:16
inc0btw..do we have R release name?21:17
sean-k-mooneyisnt the next one Q not R21:17
inc0ahh21:17
inc0well, in Polish we don't use Q, so my bad21:17
sean-k-mooneyi think Queens was proposed.21:18
*** openstackgerrit has quit IRC21:18
sean-k-mooneyinc0: no worries if i was going by the irish alphbet we only have 18 letters instad of 2621:18
sean-k-mooneyits by everyone get confused when irish names with bh are pernouced like v21:19
kfox1111back.21:19
inc0in Polish we have 32 letter, but V,Q,X are not in it21:20
inc0we have couple polish only letter which taught me a lot about encoding in software21:20
sean-k-mooneyhaha ya if you go outside of the ascii range it gets more diffucult utf-8 is interesting...21:23
kfox1111yeah, I like linux's general, just use utf8 solution over window's, lets support 30,000 different types of data types for unicode. ;)21:24
inc0my woe was web browsers21:25
inc0any python 2.x is all ascii21:25
*** dturner has joined #openstack-kolla21:26
*** athomas has quit IRC21:26
sean-k-mooneykfox1111: well windows devs generally use utf8 now too21:28
sean-k-mooneythe issue is had was how to store utf8 perperly when each charater can be 1-4 bytes and may or may not contaien a byte order mark.21:29
*** l4yerffeJ has quit IRC21:34
*** Jeffrey4l has quit IRC21:34
*** JoseMello has quit IRC21:35
sean-k-mooneyby the way anyone heard of or used ansible-lint https://github.com/willthames/ansible-lint21:36
sean-k-mooneyit might be worth looking into adding for kolla-ansible21:37
*** gfidente has quit IRC21:39
*** srwilkers has quit IRC21:43
*** fguillot has quit IRC21:44
*** rhallisey has quit IRC21:44
dmsimardsean-k-mooney: huh, interesting bit about the 18 letters alphabet .. explains a couple pronounciations indeed21:47
dmsimardlike Eoghan21:48
inc0yeah, his name is crazy, when someone pronounced it to me, I had no idea that he's he21:49
sean-k-mooneywe have a foda over vouels too, áéíóú but they are not considered seperate leters21:49
kfox1111seeing some weird errors.... http://logs.openstack.org/82/419082/9/check/gate-kolla-kubernetes-deploy-centos-binary-external-ovs-nv/f8ff995/logs/openstack/kolla-neutron-server-1133147973-nxsz0.txt21:49
sean-k-mooneyowen is the english speeling of Éoghan21:50
*** openstackgerrit has joined #openstack-kolla21:50
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: Set versions to 0.4.0 and 3.0.2  https://review.openstack.org/41908221:50
inc0now I know that21:50
*** jrobinson is now known as jrobinson-afk21:50
inc0but didn't at the time21:50
openstackgerritsayantani proposed openstack/kolla: Improve Quick Start documentation  https://review.openstack.org/40499321:50
sean-k-mooneyya i used to use english phontic spelling in irish. it did not work very well21:51
sean-k-mooneyespcially when i used letters that did not exist21:52
dmsimardlol21:52
kfox1111heh. you know you have a big company when you read "amazon" instead of the real word in the article, "arizona" over and over... :/21:54
sean-k-mooneykfox1111: i have dyslexia and if my brain half reads "arizona" and desides its "amazon" it will be so until i go do something else and then come back to it21:56
sbezverkkfox1111: any word on that global api issue I hit? I think it is still might be out there..21:56
kfox1111sbezverk: let me look again.21:56
sbezverkkfox1111: I would appreciate, probably I will not see it again with new variables in values.yaml, but I am afraid it will hit us in different place21:57
sean-k-mooneydid that post in the channel or just set my away status locally21:59
*** jtriley_ has joined #openstack-kolla22:03
*** inc0 has quit IRC22:04
*** jtriley has quit IRC22:06
*** absubram has quit IRC22:07
*** jtriley_ has quit IRC22:08
kfox1111arg... the docker hub is still messed up. :/22:15
kfox11112017-01-12 22:11:51.128735 | Tag 3.0.1 not found in repository docker.io/kolla/ubuntu-binary-memcached22:15
*** unicell has quit IRC22:16
*** salv-orl_ has joined #openstack-kolla22:23
sbezverkkfox1111: are you sure we did not use 2.0.2 before? maybe 3.0.1 has not been ever pushed for ubuntu22:24
*** mannamne has quit IRC22:25
*** unicell has joined #openstack-kolla22:25
*** salv-orlando has quit IRC22:25
*** msimonin has joined #openstack-kolla22:27
dmsimardkfox1111: I had issues with docker hub yesterday, are they still having problems ?22:28
dmsimardI would push and things would not show up.22:29
*** mannamne has joined #openstack-kolla22:29
kfox1111sbezverk: possible.22:33
kfox1111dmsimard: yeah, I think i've still seen some flakeyness on pulls.22:33
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes: Glance service orchestration with entrypoint  https://review.openstack.org/41073622:38
*** lrensing has quit IRC22:39
*** goldyfruit has joined #openstack-kolla22:40
sbezverkkfox1111: have a second for qq?22:51
*** mannamne has quit IRC22:56
*** mannamne has joined #openstack-kolla22:56
dmsimardkfox1111: oh well there you go http://status.docker.com/23:01
sdakedmsimard thats pretty cool - thanks for the link :)23:04
*** pramodrj07 has joined #openstack-kolla23:04
sdakekfox1111 this is why we need to get away from the docker hub as part of our ci infra :)23:04
sdakekfox1111 we need to be building images in the gate itself23:04
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes: Glance service orchestration with entrypoint  https://review.openstack.org/41073623:06
kfox1111sdake: I don't disagree. but building images per run is not good either.23:06
sdakethis is how kolla-ansible works23:07
sdakeits ottally fine23:07
sdakeit sometimes fials23:07
kfox1111sdake: it takes 20 min per gate run already. doubling that by uilding containers would really cut down on productivity. we need a cache.23:07
sdakerecheck ftw23:07
kfox1111sbezverk: yeah. but, do you have a link to that previous running job of yours?23:07
*** PramodJ has quit IRC23:07
sdakeya - 40 min gate times are what we have currently for kolla-ansible23:07
*** MasterOfBugs has quit IRC23:07
kfox1111I was trying to watch it to see, but you rechecked to quick. :/23:07
kfox1111sdake: I'd be much less productive witha 40 min gate. :/23:07
sdakekfox1111 i'd challeng ethat productivity is busted when docker hub is down23:07
sdakekfox1111 and docker hub is super unreliable23:08
sbezverkkfox1111: ok we will deal then when it hits again23:08
kfox1111sdake: yeah, but the hub's been down much less then 20 extra min * dozens of ps's updates a day.23:08
*** MasterOfBugs has joined #openstack-kolla23:08
sbezverkkfox1111: by some reason glance manage db job did not have init contrainer at all? any particular reason why it was removed?23:08
sdakeright - the wya i deal with gate lag is to work on multiple things at once23:08
sdakeit requires some context switching but its not too terrible23:08
kfox1111sdake: that, I haven't seen. its been very reliable. probably down 3 days out of all of my time working on the kolla-kubernetes gate.23:09
kfox1111yeah. I do that too. but then you have merge conflict overhead to deal with.23:09
sdakethe other problem is we aren't testing the latest kolla master containers ;)23:09
kfox1111there's a lot of tradeoffs.23:09
kfox1111really we need to cache 3.0.2 on tarballs.o.o.23:09
sdakeya - merge conflict overhead is a thing but not that major23:09
kfox1111or a registry.23:09
kfox1111then we can keep the 20 min gates.23:09
sdakeinc0 is working on that in some way to speed up the gate23:09
kfox1111yup.23:09
kfox1111he's really close too.23:10
sdakehowever in the short term i think we need to transition to building images on each gate run23:10
kfox1111procrastination will solve this problem. ;)23:10
sdakedefinately before we release 1.0.0 we need to be building againset master openstack23:10
kfox1111I think his trunk cache is close. much better to just wait for that.23:10
sdakethat way we can select the images we use23:10
kfox1111those images are tested too. so more likely to be reliable.23:10
sdakewe can wait on that sure - not sure where it is implementation wise23:10
*** Pavo has joined #openstack-kolla23:10
kfox1111he's got a ps half done to make kolla-kubernetes use the cache.23:11
kfox1111I was going to help with that once 0.4.0's done.23:11
*** mannamne has quit IRC23:11
sdakejust pointing out that testing with newton images is less then ideal when kolla-kube will implement ocata ;)23:11
kfox1111we'll always be chacing a newer version. ;)23:11
kfox1111ops usually only care about a stable version though. ;)23:11
kfox1111we need to ensure both work.23:12
kfox1111but if you want to make your customers happy, stable's got to work well.23:12
kfox1111chacing trunk just lets you get to the new stable sooner.23:12
kfox1111as you find issues quicker and fix them incrementally rather then all at once.23:13
*** Pavo has quit IRC23:13
kfox1111so....23:14
kfox11113.0.1 everything with 3.0.2 for a few things passed the gate for the things that didn't have hub issues.23:15
kfox1111but 3.0.2 everything was failing.23:15
kfox1111well, for centos.... ubuntu seems to be missing some 3.0.1 packages. :/23:15
kfox1111what a mess.23:16
*** Pavo has joined #openstack-kolla23:17
*** dave-mccowan has quit IRC23:17
kfox1111so guys, what do you want to do?23:18
kfox11112.0.2 based gate is working well.23:18
sbezverkkfox1111: staying with 2.0.2 is not an option23:19
kfox1111a 3.0.1 mostly version coudl be made to work too.23:19
kfox1111a 3.0.2 could be made to work with more debugging and maybe some new containers.23:19
*** saneax-_-|AFK is now known as saneax23:19
kfox1111its a snapshot in time release, so we can fix things in a 0.5.0 whenever we want to do that.23:20
kfox1111so, do we delay 0.4.0 further?23:20
sbezverkkfox1111: personnaly I could not care less about 0.4.0, but I do care about 3.0.223:23
sbezverkmaybe we could get gate stable on 3.0.2 and then release 0.4.0?23:24
kfox1111I need 0.4.0 soonish but can probaby wait till next week. I need to start deploying it and would rather not deploy trunk.23:24
kfox1111I think thats probably doable in the next few days.23:25
kfox1111just gota figure out whats different between 3.0.1 and 3.0.2.23:25
kfox1111I can try pinning neutron back down to 3.0.1 and leave the rest at 3.0.2 and see if that fixes things. it only seemed to be neutron being affected.23:25
*** Pavo has quit IRC23:26
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: Set versions to 0.4.0 and 3.0.2  https://review.openstack.org/41908223:27
sbezverksure, then instead of fighting with everything we can deal with just neutron23:27
sbezverkreleasing 0.4.0 now I think is not right because of some instability we see with newer images/cintainers23:28
kfox1111yeah, thats kind of the problem with using untested containers though. you can never really know if it will work. :/23:29
kfox1111thats the real power of docker.23:29
kfox1111you can use the exact container that you gate tested against.23:29
kfox1111one less thing that can go wrong.23:29
sbezverksure if docker hub is still alive ;-)23:30
kfox1111you dont' run into stuff like, "building ubuntu container on centos 7.3.2 builds broken containers"23:30
kfox1111true. but there are other ways around that. :)23:30
*** aj_power has joined #openstack-kolla23:30
kfox1111you just make sure you have containers on the hub and quay or wherver.23:31
*** l4yerffeJ has joined #openstack-kolla23:31
kfox1111replication is relatively easy. :)23:31
kfox1111building binary identicle containers is tough.23:31
*** Pavo has joined #openstack-kolla23:34
*** haplo37_ has quit IRC23:36
*** Jeffrey4l has joined #openstack-kolla23:37
kfox1111sbezverk: don't push antyhing yet.23:38
kfox1111looking at your ps.23:38
*** haplo37_ has joined #openstack-kolla23:39
kfox1111sbezverk: there is a one character misalignment in values.yaml.23:40
*** Pavo has quit IRC23:40
kfox1111sbezverk: but, since all your values are set to true,23:41
kfox1111you can just make that values file:23:41
kfox1111globals:23:41
kfox1111  kolla:23:41
kfox1111    all:23:41
kfox1111            kubernetes_entrypoint: true23:41
kfox1111the rest of the stuff will be automatic.23:41
*** Pavo has joined #openstack-kolla23:42
*** msimonin has quit IRC23:42
*** msimonin has joined #openstack-kolla23:42
openstackgerritsayantani proposed openstack/kolla-ansible: Modify config.yml playbook for bifrost role  https://review.openstack.org/41919623:43
*** msimonin has quit IRC23:43
*** msimonin has joined #openstack-kolla23:43
*** msimonin has quit IRC23:44
*** msimonin has joined #openstack-kolla23:44
*** schwicht has quit IRC23:44
*** msimonin has quit IRC23:44
*** msimonin has joined #openstack-kolla23:45
*** msimonin has quit IRC23:45
*** msimonin has joined #openstack-kolla23:45
sayantani01sdake_ I made changes based on your comment. It will be great if you can take a last look (hopefully last :P)23:46
*** msimonin has quit IRC23:46
*** msimonin has joined #openstack-kolla23:46
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes: Glance service orchestration with entrypoint  https://review.openstack.org/41073623:47
*** msimonin has quit IRC23:47
sbezverkkfox1111: I will change it as last step23:48
sbezverkit is still failing.. but it was my error23:48
*** salv-orl_ has quit IRC23:48
sdakesayantani01 sounds good23:48
sayantani01Thanks sdake :)23:50
kfox1111Tag 3.0.1 not found in repository docker.io/kolla/ubuntu-binary-neutron-server really?23:50
kfox1111did ubuntu just not get a 3.0.1 release of a bunch of things?23:50
*** sayantani01 has quit IRC23:50
sbezverkkfox1111: that what I was trying to tell you :-) 3.0.1 ubuntu binary has not been pushed23:52
sbezverkjust some bits anf pieces23:52
sbezverkwhat if you use 3.0.2 tag? instead of 3.0.123:53
kfox1111sorry. I didn't quite get that. :/23:53
* kfox1111 sighs.23:53
kfox1111well, at least for centos, 3.0.2 doesn't seem  to work.23:53
kfox1111still digging into why.23:53
kfox1111I was suspecting neutron, so I pinned that down to 3.0.1, but ubuntu's choking on it.23:53
sbezverkkfox1111: how about pinning neutron back to 2.0.2?23:54
sbezverkmaybe doing that we could identify all failing cases and bring at least majority to 3.0.223:55
Pavoneed me to build 3.0.2 and try it for centos kfox1111?23:55
*** lamt has quit IRC23:55
kfox1111may be an option. going to see what falls out of 3.0.1 first. maybe it will tell us whats wrong with the 3.0.2 packages if anything.23:55
*** schwicht has joined #openstack-kolla23:56
Pavohow are you see what fails?23:56
kfox1111Pavo: I'm watching zuul.23:57
kfox1111status.openstack.org/zuul/23:57
Pavobecause my 3.0.1 works pretty good minus murano and magnum horizon plugins23:57
Pavoah23:57
kfox1111ps 41908223:57
Pavoits this k8s?23:58
kfox1111yeah.23:58
kfox1111k8s gate.23:58
Pavonice23:58
kfox1111the problem is, its hard to know if your 3.0.1 is closer to the hub's 3.0.1 or the hub's 3.0.2, because of when it was built. :/23:58
Pavomakes sense23:59
kfox1111ok. the good news, the global api is working very nicely. :)23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!