Tuesday, 2016-10-04

sdakesbezverk_ i can tell yo uwhy00:00
sdakenetworking performance00:00
sdakedocker proxy is slower then a 28.8k modem00:00
sbezverk_docker proxy is still running in docker and since they changed proxy to use iptables I was told that the performance imrpoved00:01
sbezverk_I meant kube proxy00:01
sdakei see00:01
sdakemaybe i'm wrong00:01
sdakebeen drinking - has happened before :)00:01
*** vinkman has quit IRC00:03
*** tonanhngo has joined #openstack-kolla00:04
*** tonanhngo has quit IRC00:05
*** tonanhngo has joined #openstack-kolla00:07
*** tonanhngo has quit IRC00:08
*** tonanhngo has joined #openstack-kolla00:08
*** HyperJohnGraham has joined #openstack-kolla00:18
*** diogogmt has joined #openstack-kolla00:20
sbezverk_kfox1111: if you need I can test part of your doc, kube integration with just ceph cluster without bells and whistels00:24
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs  https://review.openstack.org/38104100:24
kfox1111sbezverk_: k.00:25
kfox1111that would be cool. gota head out very soon though. been up since 1:30am this morning. :/00:25
sbezverk_kfox1111: ok I will reply to PS if I hit anything00:27
kfox1111k. thx.00:28
*** msimonin has joined #openstack-kolla00:28
kfox1111its such an odd thing. both ps's seem like they should be working, or really close... but things are just sketchy...00:28
kfox1111very similar stuff in minkube just works.00:28
*** root____8 has joined #openstack-kolla00:29
*** root____8 has left #openstack-kolla00:30
*** slivi323 has joined #openstack-kolla00:30
*** daneyon has joined #openstack-kolla00:32
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: Start testing the deployments  https://review.openstack.org/38086800:34
kfox1111there. I officially give up on it being ready soon. :/00:35
sdakerbergeron around?00:36
sdakekfox1111 gate changes hard - don't get discouraged :)00:36
*** daneyon has quit IRC00:36
sdakekfox1111 if its too much to tkae on in one patch split it up into multiple chunks of work00:37
sdakethats what i typically do00:37
sdaketo try to make the work less00:37
kfox1111I'm fine. just overly tired at this point.00:37
kfox1111I'm too stubbern to let the gate win. ;)00:37
kfox1111well.... there's a rax node...00:39
kfox1111and its takign forever...00:39
britthouser2I want a t-shirt that says, "I'm too stubbern to let the gate win."00:39
sdakebritthouser2 that would be winning00:40
sdakewe ought to put that on the cisco shirts ;)00:40
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes: Adding kubectl commands to setup default namespace  https://review.openstack.org/38134900:40
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs  https://review.openstack.org/38104100:41
kfox1111now I'm just making stupid mistakes. :/00:41
kfox1111k. have a good one folks. I'm going to bed. :)00:42
britthouser2Gnight kfox1111.  Sleep to fight the gate another day.00:42
sbezverk_kfox1111: take care00:42
kfox1111you too.00:44
HyperJohnGrahamhi all00:45
*** slivi323 has quit IRC00:54
*** MarMat has quit IRC00:54
*** jax3242 has quit IRC00:55
*** phuongnh has joined #openstack-kolla00:59
*** v1k0d3n has joined #openstack-kolla01:00
*** awidders_ has quit IRC01:03
*** awiddersheim has joined #openstack-kolla01:03
*** zhubingbing has joined #openstack-kolla01:05
*** zhubingbing has quit IRC01:18
*** zhubingbing has joined #openstack-kolla01:19
*** duonghq has joined #openstack-kolla01:29
*** msimonin has quit IRC01:31
*** zhubingbing has quit IRC01:35
sbezverk_HyperJohnGraham: finally rebuild my cluster and got mariadb running so tomorrow I can work with you to get yours going01:38
HyperJohnGrahamoh very cool !01:38
HyperJohnGrahami have meetings in the morning but should be free by noon pacific time01:40
*** schwicht_at_work has joined #openstack-kolla01:42
sbezverk_ok it is not the best time for me because people tend to break networks around 3PM EST ;-) but hopefully tomorrow it will not be the case01:42
*** zhubingbing has joined #openstack-kolla01:42
*** zhubingbing has quit IRC01:43
*** schwicht has quit IRC01:43
openstackgerritJeffrey Zhang proposed openstack/kolla: Add hexdump package for ironic-conductor container  https://review.openstack.org/38123801:44
*** zhubingbing has joined #openstack-kolla01:47
*** zhubingbing has quit IRC01:52
*** inc0 has quit IRC01:59
sdakecan someone let pbourke know that jewel needs to be implemented in oracle linux02:04
*** MarMat has joined #openstack-kolla02:05
*** unicell has quit IRC02:07
sdakemarmat how is magnum looking02:12
HyperJohnGrahamsbezverk_: early morning too02:19
HyperJohnGrahambut i have a 10am dentist appointment i need to break for02:20
HyperJohnGrahamping me in the am any time if you want02:21
sdakesbezverk_ my finest work is done at 3pm est ;)02:22
*** david-lyle has joined #openstack-kolla02:25
*** Pavo has quit IRC02:27
*** tonanhngo has quit IRC02:28
MarMatsdake looks fine with me... you mean anything particular?02:31
*** tonanhngo has joined #openstack-kolla02:31
*** Pavo has joined #openstack-kolla02:33
*** david-lyle has quit IRC02:38
*** david-lyle has joined #openstack-kolla02:49
*** david-lyle has quit IRC02:55
*** caowei has joined #openstack-kolla03:00
*** g3ek has quit IRC03:00
*** caowei has quit IRC03:01
*** haplo37 has quit IRC03:01
*** g3ek has joined #openstack-kolla03:07
*** haplo37 has joined #openstack-kolla03:07
*** unicell has joined #openstack-kolla03:13
*** daneyon has joined #openstack-kolla03:14
*** daneyon has quit IRC03:19
*** eaguilar has quit IRC03:21
sdakemarmat is it good to ship03:31
MarMatsdake subjectivey so far ok, still running tests on newest builds and different configurations... are there any criteria to be "good to ship"?03:35
sdakemarmat a) does it work is criteria 103:37
sdakeb) does it work well is criteria 203:37
sdakeif those ren't met, then we releasenote it as techpreview03:38
sdakeat ummit we will come up with a better mechanism for handling what we consider the inner core, outer core, and extra outer core03:38
sdakeor whatever03:38
sdakewho knows :)03:38
sdakeactually if a is met, we dont need a techpreviewe releasenote03:39
MarMatsdake well basic configuration (no barbican, no lb, cinder lvm) is working. More complex configurations either need some workarounds or we have not tested yet.03:42
sdakewould be curious to know if cinder ceph works03:42
sdakethat would be good enough for me03:42
MarMatsdake there are still some weeks for decision, right?03:43
sdake8 days03:43
MarMatsdake ask me then in 8 days :-)03:43
MarMatsdake is anyone else testing magnum?03:44
sdakei used to03:44
sdakebut no time now03:44
sdakeso your our go to on that decision :)03:44
sdakenote its the cinese national holiday so we are missing some of our ch contribs03:45
sdakei think for a week :(03:45
sdakeJeffrey4l__ still seems to be at it03:45
sdakebut iiuc he is our highest output contributor03:45
Jeffrey4l__sup03:45
sdakeJeffrey4l__ not a ping just mentioning your name03:46
Jeffrey4l__Got ;)03:46
MarMatsdake roger that, will proceed with some more magnum tests.03:46
sdakecinder + ceph would be good03:47
HyperJohnGraham_:)03:47
sdakeits too late this cycle to set quality standards for components03:47
sdakewe know our core is fantastic03:47
sdakeand the outer core (things like heat/horizon/etc) work fantastic03:47
sdakethe rest of the added services - unknown03:48
MarMatsdake talking about quality, a question - how it came that the rabbit bump issues from Friday were not detected by CI/checks/gates?03:50
sdakethey were03:50
sdakethe gate was down fri-mon03:50
sdakeearly am my time03:50
sdakebut our ci checks are weak03:51
sdakethey need alot of attention03:51
MarMatI see03:51
*** yuanying has quit IRC04:01
*** yuanying_ has joined #openstack-kolla04:01
*** jmccarthy has quit IRC04:03
*** jmccarthy has joined #openstack-kolla04:04
openstackgerritMerged openstack/kolla: Revert "Fix pycparser wheel package issue"  https://review.openstack.org/38123704:06
openstackgerritMerged openstack/kolla: Add hexdump package for ironic-conductor container  https://review.openstack.org/38123804:07
openstackgerritMerged openstack/kolla: install pxelinux for ironic_pxe  https://review.openstack.org/38035004:07
*** absubram has joined #openstack-kolla04:17
*** absubram_ has joined #openstack-kolla04:18
*** absubram has quit IRC04:21
*** absubram_ is now known as absubram04:21
*** Pavo has quit IRC04:27
*** salv-orlando has joined #openstack-kolla04:29
*** lamt has quit IRC04:31
*** Pavo has joined #openstack-kolla04:32
*** salv-orlando has quit IRC04:34
*** senk__ has joined #openstack-kolla04:39
*** senk__ has quit IRC04:39
openstackgerritOpenStack Proposal Bot proposed openstack/kolla: Updated from global requirements  https://review.openstack.org/37598905:03
*** unicell1 has joined #openstack-kolla05:04
*** unicell has quit IRC05:05
*** salv-orlando has joined #openstack-kolla05:07
*** MarMat has quit IRC05:12
bjolo_morning05:21
bjolo_sdake, still up? ok to build or still issues?05:24
*** unicell1 has quit IRC05:29
*** unicell has joined #openstack-kolla05:30
*** g3ek has quit IRC05:34
*** haplo37 has quit IRC05:34
*** g3ek has joined #openstack-kolla05:39
*** msimonin has joined #openstack-kolla05:40
*** haplo37 has joined #openstack-kolla05:40
*** flaper87 has joined #openstack-kolla05:49
*** flaper87 has quit IRC05:49
*** flaper87 has joined #openstack-kolla05:49
*** daneyon has joined #openstack-kolla05:57
*** daneyon has quit IRC06:01
*** Jeffrey4l__ has quit IRC06:14
*** MartinsG has quit IRC06:14
*** Jeffrey4l__ has joined #openstack-kolla06:14
openstackgerritgecong proposed openstack/kolla-kubernetes: Replace LOG.warn with LOG.warning  https://review.openstack.org/38142906:14
*** Jeffrey4l__ has quit IRC06:14
*** Jeffrey4l_ has joined #openstack-kolla06:15
*** Jeffrey4l_ is now known as Jeffrey4l06:15
*** salv-orlando has quit IRC06:21
*** msimonin has quit IRC06:24
*** Pavo has quit IRC06:27
openstackgerritTakashi Sogabe proposed openstack/kolla-kubernetes: A fix of tunnel_interface in neutron-openvswitch-agent  https://review.openstack.org/38143406:29
*** Pavo has joined #openstack-kolla06:34
sdakebjolo build is good to go06:36
sdakebjolo yup still up just got home from folks house06:36
sdakebjolo headed to bed tho - night06:36
jmccarthyHi all !06:39
jmccarthyCeph question if anyone knows, why is fstab updated with info to mount disks on the hosts ?06:39
jmccarthyAren't these mounted directly by the containers for themselves anyway ?06:40
*** egonzalez90 has joined #openstack-kolla06:40
*** tonanhngo has quit IRC06:46
*** unicell1 has joined #openstack-kolla06:50
*** unicell has quit IRC06:51
*** tonanhngo has joined #openstack-kolla06:53
*** hieulq has joined #openstack-kolla06:54
*** tonanhngo has quit IRC06:54
openstackgerritMerged openstack/kolla: Change mysql-check in HAProxy to post MySQL 4.1  https://review.openstack.org/38121807:01
*** senk_ has joined #openstack-kolla07:04
*** senk_ has quit IRC07:06
*** ankush has joined #openstack-kolla07:09
*** salv-orlando has joined #openstack-kolla07:11
duonghqanybody see zhubingbing around?07:14
*** matrohon has joined #openstack-kolla07:14
*** msimonin has joined #openstack-kolla07:14
bjolo_jmccarthy, dont have kolla ceph deployed myself. Does the containers mount /etc/fstab from the host?07:16
*** tonanhngo has joined #openstack-kolla07:16
*** tonanhngo has quit IRC07:19
jmccarthybjolo_ I'd have to look, I'm not that familiar with the implementation actually, I'm think of swift (which does not modify fstab and seems to work just fine)07:24
*** b_bezak has joined #openstack-kolla07:27
*** salv-orlando has quit IRC07:33
*** salv-orlando has joined #openstack-kolla07:46
*** hieulq has quit IRC07:47
*** Serlex has joined #openstack-kolla07:51
duonghqsdake: around?07:51
*** g3ek has quit IRC07:58
*** haplo37 has quit IRC07:59
*** imcsk8 has quit IRC08:01
*** imcsk8 has joined #openstack-kolla08:01
sdakeduonghq yess just responding to an email08:01
sdakeand ready to rack08:01
sdakewhats up08:01
duonghqsdake: for the precheck task08:02
duonghqcurrently, we check through ping08:02
duonghqso, we don't cope with the blocked ICMP08:03
duonghqor we haven't coped with it08:03
*** hieulq has joined #openstack-kolla08:03
*** g3ek has joined #openstack-kolla08:05
duonghq(I mean the kolla_internal_vip_address check)08:05
*** haplo37 has joined #openstack-kolla08:06
*** salv-orlando has quit IRC08:12
sdakeright08:13
duonghqsdake: so, when we cannot ping the ip, can we assume the IP does not existed08:15
*** athomas has joined #openstack-kolla08:15
*** salv-orlando has joined #openstack-kolla08:19
*** salv-orlando has quit IRC08:21
*** salv-orlando has joined #openstack-kolla08:21
sdakeduonghq right08:26
sdakeduonghq have proposal to fix it?08:26
sdakeduonghq i think its such a low priority issue at this point not worth disturbing the release over08:26
sdakebut please file a bug08:26
sdakeand make a case :)08:26
*** msimonin has quit IRC08:27
*** Pavo has quit IRC08:27
*** hieulq has quit IRC08:28
duonghqsdake: roger08:30
*** msimonin has joined #openstack-kolla08:31
*** Pavo has joined #openstack-kolla08:32
*** hieulq has joined #openstack-kolla08:33
duonghqsdake: it's really a small bug but I heard that in many case, org block icmp from switch/router manner?08:34
*** tonanhngo has joined #openstack-kolla08:35
*** tonanhngo has quit IRC08:36
sdakerbergeron you around08:39
sdakeduonghq - would like to hear a suggesteddworkaround08:40
sdakeif you can give me a 45 second ptich that would rock08:40
sdakeelse i am off to bed08:40
duonghqsdake: I'll check ssh port08:40
duonghqbecause Ansible work through ssh08:40
duonghqso check if the ssh open is quite resonable?08:41
sdakeworks for me08:41
duonghqthank sdake, so now you hit the bed?08:42
sdakebetter then ping imo08:42
sdakealthough ansible ping usess ssh :)08:42
sdakemaybe we are using ping directly which is a problem08:42
duonghqsdake: ya, I'll rewrite that,08:42
duonghqI'm always wonder why we use raw ping08:43
duonghqso I'll switch to ping module08:43
duonghqhope that it's good enough08:43
*** quasar has joined #openstack-kolla08:49
*** tonanhngo has joined #openstack-kolla08:54
*** tonanhngo has quit IRC08:57
sdakerbegeron ping09:02
sdakeduonghq ya ping module uses ssh pretty sure09:02
duonghqsdake: the ping module is not designed to use in playbook09:03
duonghqso wait_for is under duty09:03
*** strigazi_AFK is now known as strigazi09:04
sdakehmm i can't say i understand why09:04
sdake(the ping module in playbooks)09:04
sdakeanyway - time to crash sorry dude ;)09:05
duonghqokay sdake, see you tommorrow09:05
*** mnasiadka has joined #openstack-kolla09:10
*** gfidente has joined #openstack-kolla09:11
*** jrich523 has quit IRC09:23
*** jrich523 has joined #openstack-kolla09:26
coolsvapduonghq: can you point me to the review in discussion?09:32
*** daneyon has joined #openstack-kolla09:33
duonghqcoolsvap: I'm working on the ps, but the bug is https://bugs.launchpad.net/kolla/+bug/163016509:34
openstackLaunchpad bug 1630165 in kolla "Improve VIP existence check" [Wishlist,New] - Assigned to Duong Ha-Quang (duonghq)09:34
duonghqcoolsvap: it's quite interesting case, indeed09:34
coolsvapduonghq: so what you intend to use?09:34
duonghqI intend use wait_for module09:35
duonghqI check for ssh_port open09:36
coolsvapwait_for is fine, so do you have a test system where you can test the exact scenario you depicted in bug?09:36
*** narasimha_SV has joined #openstack-kolla09:37
coolsvapI have seen wait_for suffers from regression while checking for ssh09:37
duonghqsure, it's quite easy to create testbed09:37
coolsvaphave seen that earlier, but we can always test it again09:37
narasimha_SVhttp://paste.openstack.org/show/584146/ while dpeloyiing keystone contianer is failing is with this error09:37
duonghqbut the problem is I cannot determine with port is used for ssh in the VIP09:37
duonghq*with -> which09:38
*** daneyon has quit IRC09:38
coolsvapi suppose you can start with default and improvise for different ports in next iteration09:43
duonghqcoolsvap: sure, so I'll try with 22, due to there is no clue for other port09:43
*** shardy has joined #openstack-kolla09:47
coolsvapduonghq: sure09:50
*** gfidente has quit IRC09:51
*** gfidente has joined #openstack-kolla09:53
openstackgerritDuong Ha-Quang proposed openstack/kolla: Improve VIP existence check  https://review.openstack.org/38158910:00
*** zigo has quit IRC10:01
*** zigo has joined #openstack-kolla10:04
*** zigo is now known as Guest8478010:05
*** senk_ has joined #openstack-kolla10:11
*** senk_ has quit IRC10:12
bjolo_hi all10:14
*** msimonin has quit IRC10:14
bjolo_question about reviews and merged code10:14
bjolo_im waiting for this PS https://review.openstack.org/#/c/373455/10:14
*** Guest84780 has quit IRC10:14
bjolo_it says merged, but i dont see the code when i do a git clone10:14
pbourkebjolo_: that change is merged10:14
*** senk_ has joined #openstack-kolla10:15
pbourkehttps://github.com/openstack/kolla/commit/d1673ad173ea96c0e018946f54caa99776855cc110:16
bjolo_ah i need to update globals myself10:16
bjolo_the start.yml code is there10:16
*** zigo_ has joined #openstack-kolla10:16
bjolo_nevermind my tired head :)10:17
*** msimonin has joined #openstack-kolla10:17
*** msimonin has quit IRC10:17
*** senk_ has quit IRC10:21
*** Pavo has quit IRC10:27
*** duonghq has quit IRC10:27
*** g3ek has quit IRC10:28
*** haplo37 has quit IRC10:28
*** ntpttr has quit IRC10:30
*** srwilkers has quit IRC10:30
*** ntpttr has joined #openstack-kolla10:31
*** Pavo has joined #openstack-kolla10:32
*** g3ek has joined #openstack-kolla10:34
*** phuongnh has quit IRC10:35
*** haplo37 has joined #openstack-kolla10:35
*** salv-orlando has quit IRC10:42
*** coolsvap is now known as coolsvap_11:01
*** senk_ has joined #openstack-kolla11:10
*** bjolo_ is now known as bjolo11:10
bjoloquestion about horizon and policy.json files11:15
*** rhallisey has joined #openstack-kolla11:15
bjolomy understanding is that horizon needs to have the same policy file as keystone has11:16
bjolobut today there is no copy config for policy files for horizon11:16
bjolowould that be deemed a bug and fixed in newton, or considered new feature and moved to ocata?11:17
*** zigo_ is now known as zigo11:23
*** haplo37_ has quit IRC11:33
pbourkebjolo: depends what the result is having them the same11:35
*** haplo37_ has joined #openstack-kolla11:35
*** narasimha_SV has quit IRC11:39
*** salv-orlando has joined #openstack-kolla11:41
openstackgerritMerged openstack/kolla-kubernetes: A fix of tunnel_interface in neutron-openvswitch-agent  https://review.openstack.org/38143411:44
*** dwalsh has joined #openstack-kolla11:45
bjolopbourke, in my use case im trying to get keystone policy.v3cloudsample.json working11:46
*** salv-orlando has quit IRC11:46
bjolonot suceeded all the way yet, but one part is to make horizon work with domains, and for that i need to copy the policy file into the container. no biggi to do manually but i would be nicer to have kolla doing it11:48
openstackgerritMerged openstack/kolla-kubernetes: Replace LOG.warn with LOG.warning  https://review.openstack.org/38142911:48
bjolocan any core reviewer please do a triage on this one? https://bugs.launchpad.net/kolla/+bug/162699511:50
openstackLaunchpad bug 1626995 in kolla "neutron_dvr fails to deploy" [Undecided,New]11:51
pbourkebjolo: hmm I think I've seen this11:53
bjoloshould it be removed or completed?11:55
bjoloseems like the code is kinda cut-off11:55
pbourkecan you post the deploy output with -vvv11:56
bjolook11:56
*** ppalacios has joined #openstack-kolla12:00
*** tonanhngo has joined #openstack-kolla12:01
*** msimonin has joined #openstack-kolla12:01
*** tonanhngo has quit IRC12:02
*** rodrigo_pereira has joined #openstack-kolla12:02
ppalaciosAm I missing something or images 2.0.2 are not properly configured? I'm trying to "kolla-ansible deploy" and when I reach the Horizon playbook, the task "creating the member role" fails after 10 tries. It looks like that the ansible within this image are missing extra modules. Does anybody can help?12:12
pbourkeppalacios: please pastebin the full log preferably with -vvv12:13
*** zhubingbing has joined #openstack-kolla12:15
*** daneyon has joined #openstack-kolla12:16
*** srwilkers has joined #openstack-kolla12:16
*** portdirect has quit IRC12:20
ppalaciospbourke: just a sec (I'm running it again)12:20
*** daneyon has quit IRC12:21
*** Pavo has quit IRC12:27
*** schwicht_at_work has quit IRC12:28
*** salv-orlando has joined #openstack-kolla12:28
openstackgerritMatthieu Simonin proposed openstack/kolla: Allow custom Type value in heka decoders.  https://review.openstack.org/38176412:32
*** Pavo has joined #openstack-kolla12:32
*** tonanhngo has joined #openstack-kolla12:34
*** tonanhngo has quit IRC12:36
*** fguillot has joined #openstack-kolla12:37
*** msimonin1 has joined #openstack-kolla12:38
*** msimonin has quit IRC12:40
*** senk_ has quit IRC12:42
bjolopbourke, sorry taking a while. i had ironic enabled as well, and it seems ironic fails to deploy also12:47
bjolopbourke, http://paste.openstack.org/show/584192/12:56
*** shardy is now known as shardy_mtg12:56
bjolothe bug is updated as well with the log paste12:56
pbourkethanks bjolo looking into it now13:00
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes: Fixing TUNNEL_INTERFACE in neutron dhcp agent pod  https://review.openstack.org/38178513:00
kfox1111monring.13:02
bjologood afternoon kfox111113:02
kfox1111:)13:02
bjolo:)13:02
britthouser2Hey pbourke - last night sdake said something about asking you about oracle linux and jewel (ceph version)13:02
britthouser2did you get the message already?13:03
bjolois etcd needed for kuryr?13:03
bjoloin globals.yml13:03
bjolokolla-ansible13:04
pbourkebritthouser2: no13:05
pbourkebritthouser2: may have been around an issue myself and a few others were seeing in ceph recently13:05
*** fguillot has quit IRC13:08
rhalliseykfox1111, hey :)13:09
*** dave-mccowan has joined #openstack-kolla13:11
ppalaciospbourke: I got it solved here. I was trying to deploy with images from https://hub.docker.com/u/kolla/ . After building the images by myself I was able to complete all playbooks. So, the remaning question is, should I always build the images by myself?13:11
pbourkeppalacios: yes it's generally the best idea. currently we dont have a good mechanism of keeping the dockerhub builds up to date13:11
kfox1111rhallisey: hey. :)13:13
sbezverk_good morning13:14
kfox1111sbezverk_: good monring. :)13:14
ppalaciospbourke: thanks!13:15
sbezverk_kfox1111: ceph-rbd purpose is just to have ceph-common and distribute ceph keyring, correct?13:15
*** schwicht has joined #openstack-kolla13:15
kfox1111sbezverk_: no. its a way not to have to install ceph on the hosts.13:15
sbezverk_kfox1111: that is what I meant :-)13:16
kfox1111it distributes a payload of a /usr/bin/rbd script that runs rbd in docker.13:16
*** egonzalez90 has quit IRC13:16
kfox1111sthe ceph conf and key still come through normally.13:16
kfox1111I did tweak the instructions though so that you deploy the ceph key via secret though.13:17
britthouser2I tried to find it in my scrollback pbourke, but not seeing it right off.13:17
sbezverk_kfox1111: I was looking into that, since we already have secret, why not to use it..13:18
kfox1111I'm working on getting rid of the admin key. its dangerous.13:19
kfox1111so the ceph-acmin container and key can be completely deleted from kubernetes once the volumes/users are inited.13:19
kfox1111ceph-admin13:20
*** tonanhngo has joined #openstack-kolla13:20
*** tonanhngo has quit IRC13:20
kfox1111the gate check sets up a kolla ceph user for the rbd volumes.13:20
pbourkebritthouser2: no worries thanks anyway13:20
bjolosdake, are you up?13:21
sbezverk_kfox1111: it sounds a little as a overcomplication at this stage, but that does not stop you ;-)13:22
kfox1111sbezverk_: As always, I'm interested in prduction worthyness. :)13:22
bjolocan i please get triage on this one https://bugs.launchpad.net/kolla/+bug/162695813:22
openstackLaunchpad bug 1626958 in kolla "neutron_server not working when lbaas enabled" [Undecided,New]13:22
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs  https://review.openstack.org/38104113:23
kfox1111almost there I think... just gota figure out the automation for formating the volumes.13:23
pbourkebjolo: still on it13:23
sbezverk_kfox1111: btw without --image-feature layering I could not attach from kube to ceph volume, so I think it is a mandatory..13:24
kfox1111sbezverk_: could be. this is hammer though. that doesn't work with hammer.13:24
kfox1111once we get a 3.0.0 set of containers released, I'll add a conditional in there for that.13:25
*** ccesario has joined #openstack-kolla13:26
sbezverk_kfox1111: ok13:26
sbezverk_rhallisey: I added kubectl commands to quick start so you do not have to type --namespace=kolla, could you please test it in your env?13:27
rhalliseyok cool13:27
sbezverk_rhallisey: https://review.openstack.org/38134913:28
kfox1111sbezverk_: keepalived seems to not quite be working right: http://logs.openstack.org/68/380868/21/experimental/gate-kolla-kubernetes-deploy-centos-binary-ceph-nv/aaa7983/console.html13:29
kfox1111curling the cluster ip works, but the external ip times out.13:29
sbezverk_kfox1111: hm, I have it running in multinode and it works great.13:31
kfox1111yeah. probably just misconfigured?13:32
*** schwicht has quit IRC13:33
sbezverk_kfox1111: can you add ping of VIP after keepalived starts?13:34
kfox1111http://logs.openstack.org/68/380868/21/experimental/gate-kolla-kubernetes-deploy-centos-binary-ceph-nv/aaa7983/logs/ip.txt13:34
kfox1111doesn't apear to be attached.13:34
*** janem has joined #openstack-kolla13:34
sbezverk_also getting logs from keepalived container would be helpfull13:34
kfox1111not much in the logs: http://logs.openstack.org/68/380868/21/experimental/gate-kolla-kubernetes-deploy-centos-binary-ceph-nv/aaa7983/logs/pods/kolla-keepalived-xbcm7-main.txt13:35
*** tonanhngo has joined #openstack-kolla13:36
*** tonanhngo has quit IRC13:37
*** ccesario has quit IRC13:37
*** inc0 has joined #openstack-kolla13:38
sbezverk_kfox1111: can you check and paste configmap generated?13:40
*** skramaja has quit IRC13:41
kfox1111one of the few things I'm not capturing yet. :/ will have to do another run.13:42
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes: Adding kubectl commands to setup default namespace  https://review.openstack.org/38134913:47
rhalliseysdake, hey13:47
rhalliseyor inc013:47
rhalliseycan you define trivialfix13:47
*** schwicht has joined #openstack-kolla13:47
inc0I don't think there is definition13:48
rhalliseycan you define when to use it13:48
rhalliseythis has confused me13:48
rhalliseyis it purely for docs? Or is it meant for things that are trivial fixes13:48
kfox1111rhallisey: "I'll know it when I see it" ? :)13:49
inc0no, it's not meant only for docs13:49
inc0it's meant for one liners which are obvious and so onm13:49
rhalliseyok that's what I see it as too13:49
kfox1111k13:50
inc0so you won't need to publish bug to every small improvement13:50
inc0lower cost of improvement = more improvements13:50
rhalliseykfox1111, reason I'm asking is beacuse it's sort of an indicator that subsitutes bugs/bps13:50
*** zhubingbing has quit IRC13:50
rhalliseyjust want to be sure we're tracking the best way13:50
inc0rhallisey, we should slowly limit it's ussage in kolla, but kolla-k8s can go ahead13:50
inc0it's a tradeoff - with it it's harder to put change into context13:51
*** sean-k-mooneyAFK is now known as sean-k-mooney13:51
inc0which is important long run when core team will stop handling following every workstream in community13:51
inc0which is kinda happening now:)13:51
wznoinskhi all13:54
rhalliseyinc0, agreed13:54
*** senk_ has joined #openstack-kolla13:54
kfox1111yeah13:54
openstackgerritOpenStack Proposal Bot proposed openstack/kolla: Updated from global requirements  https://review.openstack.org/37598913:54
wznoinskwould anyone have some experience with iscsid container? asking because I'm planning to use it as dependency for ironic-conductor instead of having iscsid in conductor's container - I can't see any testing of this hence would need somebody to eyeball my proposal13:55
*** egonzalez90 has joined #openstack-kolla13:56
*** ccesario has joined #openstack-kolla13:57
*** dwalsh has quit IRC13:57
*** MarMat has joined #openstack-kolla13:58
*** dwalsh has joined #openstack-kolla13:58
*** diogogmt has quit IRC13:59
*** coolsvap_ is now known as coolsvap13:59
*** dwalsh has quit IRC14:00
pbourkeinc0: hey inc0, just wondering did you ever get round to that ceph too many pgs bug?14:00
pbourkeinc0: if not I can look at it14:00
-openstackstatus- NOTICE: The Gerrit service on review.openstack.org is being restarted to address performance degradation and should return momentarily14:01
inc0pbourke, could you? I really need to get myself into kolla-k8s;)14:01
*** dwalsh has joined #openstack-kolla14:01
pbourkeinc0: np14:01
inc0I'm to present on summit about it after all;)14:01
inc0let me know if you need any help14:01
inc0also Sean's mons are working14:01
*** pbourke has quit IRC14:02
*** salv-orlando has quit IRC14:02
*** pbourke has joined #openstack-kolla14:03
*** tonanhngo has joined #openstack-kolla14:03
*** eaguilar has joined #openstack-kolla14:03
*** senk_ has quit IRC14:03
*** tonanhngo has quit IRC14:03
sean-k-mooneypbourke: i have hit that too in the past i think the main issue is that the the amount of pgs are hardcoded. it would be great if that was a config option.14:04
pbourkesean-k-mooney: yeah it needs to be fixed. Issue is last time I tried ceph master Im still getting the mon error14:05
sean-k-mooneypbourke: i was able to get past it by not using 6 mons. i was goint to look at the pg bug once i finished getting this cluster working but its probably a week out before i will have time14:06
*** shardy_mtg is now known as shardy14:06
*** matrohon has quit IRC14:06
pbourkesean-k-mooney: I use 3 mons14:06
*** dwalsh has quit IRC14:06
sean-k-mooneyusing 3 mons and removing dumb-init worked for me14:07
sean-k-mooneyi havent tested with dumb-init14:07
sbezverk_inc0: if you have not put together kube multinode, I suggest you check new kubeadm, with minor parameters change, I brought up 5 node cluster and kolla-kube runs on it..14:07
inc0sbezverk_, I've already done it:)14:07
HyperJohnGrahammorning all14:08
inc0https://github.com/inc0/kargo14:08
sean-k-mooneypbourke: i also have get a fully working cluster as heat was failing so i havent tried spawning vms but i got passed the mons hanging.14:08
pbourkeinc0: sean-k-mooney: rhallisey: anyone opposed to installing the clients in kolla-toolbox?14:08
inc0pbourke, nah, but why?14:08
pbourkeinc0: handy as a toolbox14:08
pbourkeinc0: given its name :)14:09
inc0you'd also need to put admin.sh there14:09
pbourkei have a script that uses it just haven't had time to submit it yet14:09
pbourkepassing in the vars etc.14:09
pbourkemaybe will wait and submit both together14:09
rhalliseypbourke,kolla-k8s is doing it14:10
pbourkeoh?14:10
rhalliseypbourke, they use to be in there14:10
rhalliseythere were deps of shade14:10
pbourkeyeah it has openstackclient14:10
*** jheroux has joined #openstack-kolla14:11
pbourkebut its missing some that arent deps14:11
rhalliseyya I'd say add them14:11
sean-k-mooneyrhallisey: oh sepaking of shade i hit a sade error with heat yesterday.14:11
rhalliseyreally14:11
rhalliseybug?14:11
sean-k-mooneyim rebuilding my images to see if its still hapening on master but it was complaining that it needed shade version 1.5 or newer14:12
sean-k-mooneysorry i have a seperate issue with heat the shade issue was with magnum14:12
*** gfidente has quit IRC14:13
sean-k-mooneythe tasks that setup the magnum trustee was failing. ill see if i can recreate and ill open a bug if i can14:13
*** fguillot has joined #openstack-kolla14:14
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla: Install openstack clients in kolla-toolbox  https://review.openstack.org/38187614:15
rhalliseysean-k-mooney, ok gotcha14:16
sean-k-mooneyby the way is it expect that i should be able to run a deploy after runing destroy when using ceph? do i need to relable the partitions for the osd to be bootstraped again?14:17
*** dwalsh has joined #openstack-kolla14:20
*** matrohon has joined #openstack-kolla14:21
*** MartinsG has joined #openstack-kolla14:22
*** lrensing has joined #openstack-kolla14:23
pbourkesean-k-mooney: I find they need to be relabelled14:23
pbourkesean-k-mooney: in fact yeah that is the case14:24
pbourkesean-k-mooney: destroy should prob do that14:24
pbourkesean-k-mooney: they need to be unmounted also I have a patch up for that14:24
MartinsGHi, guys! I was wondering if there are any plans on implementing SELinux policies for qemu. Without Docker libvirt has SELinux policies, which allow running each qemu-kvm in separate selinux domain and secure it.14:24
pbourkeMartinsG: none currently though I think we have a meeting this week with someone from the security group14:25
pbourkeMartinsG: so I can mention it if you like, or feel free to join and bring it up :)14:25
sean-k-mooneypbourke: what i have seen in the past is that it deploys the osd's but they stay restarting if i dont relable.14:25
sean-k-mooneypbourke: having destroy read be able to restory the original lables would be useful.14:26
*** ccesario has quit IRC14:27
*** Pavo has quit IRC14:27
*** ccesario has joined #openstack-kolla14:28
*** denaitre has joined #openstack-kolla14:28
MartinsGpbourke: Thank you! :)14:29
*** b_bezak has quit IRC14:29
pbourkesdake: inc0: was it thursday for the security meeting?14:30
inc0yeah14:30
pbourkeinc0: time?14:31
pbourke(will send a ML)14:31
inc03pm utc14:31
*** Pavo has joined #openstack-kolla14:32
sbezverk_rhallisey: ping14:33
*** tonanhngo has joined #openstack-kolla14:35
MarMatsean-k-mooney what is exactly the error with shade? I have bumped shade to 1.5.0 in kolla-toolbox because of magnum14:35
*** tonanhngo has quit IRC14:36
rhalliseysbezverk_, yes sir14:36
MarMatsean-k-mooney magnum is building fine for (for centos)14:37
rhalliseysbezverk_, I'm not opposed to your patch14:37
rhalliseybtw14:37
*** tonanhngo has joined #openstack-kolla14:37
rhalliseyI think though, there is more to be done there14:37
sbezverk_rhallisey: that part I get :-)14:37
sbezverk_it is just the idea of using user's current context for the namespace for kolla services does not sound right to me..14:38
sbezverk_but I might be wrong :-)14:39
rhalliseywhere I'd think it's ok to have 2 sources of truth here is if the op will ignore these steps because they want to stay in the deafult namespace14:40
sbezverk_rhallisey: fine in this case they will configure kolla_namespace: default and that is it14:41
rhalliseythe reason for the error checking is because it's a common mistake to not know your namespace14:41
*** senk_ has joined #openstack-kolla14:42
*** bmace has quit IRC14:42
rhalliseysbezverk_, I'm ok with merging as is and leaving kolla_namespace var in place.  I think we can revisit precheck/logging for this though14:43
sbezverk_rhallisey: idk I am afraid the error checking logic could be very complex. how do you propose detection of wrong namespace in cli?14:43
rhalliseycheck current namespace and compare to config file namespace14:44
sean-k-mooneyMarMat: it was building fine but when kolla tried to deploy it failed. i belive it was failing here https://github.com/openstack/kolla/blob/869aa073b549c837b32a8538a40f98d419ce5a84/ansible/roles/magnum/tasks/register.yml#L60-L7314:44
rhalliseynot equal just throw a one time warning14:44
sbezverk_rhallisey: ok got it..14:45
rhalliseysbezverk_, don't need to do it now14:45
rhalliseyprechecks/logging will come soon14:45
rhalliseyI'll make a bp14:45
sbezverk_rhallisey: sounds good.. :-)14:46
MarMatsean-k-mooney yep, os_user ansible module has hependency on shade 1.5 and this was added to kolla-toolbox, so should run fine.14:46
rhalliseysbezverk_, thanks :)14:46
HyperJohnGrahamsbezverk_: I have 90 minutes this morning before my first appointment. Do you have some time ?14:46
sean-k-mooneyMarMat: ya i was likely still running an old kolla_toolbox. from a week or two ago. did not do a full rebuild just a rebuild of the ceph container when i hit the issue14:47
*** lrensing has quit IRC14:47
openstackgerritMerged openstack/kolla-kubernetes: Adding kubectl commands to setup default namespace  https://review.openstack.org/38134914:51
*** salv-orlando has joined #openstack-kolla14:51
sbezverk_HyperJohnGraham: Hi, yes14:55
HyperJohnGrahamAhh cool14:55
*** unicell1 has quit IRC14:55
HyperJohnGrahamdid you want to use webex again ?14:56
sbezverk_HyperJohnGraham: yes please, here is the link: https://cisco.webex.com/join/sbezverk14:56
*** salv-orlando has quit IRC14:56
*** MartinsG has quit IRC15:01
*** egonzalez90 has quit IRC15:07
*** dave-mccowan has quit IRC15:13
*** MarMat has quit IRC15:15
*** MarMat has joined #openstack-kolla15:16
*** diogogmt has joined #openstack-kolla15:21
sean-k-mooneyis anyone familar with kollas ha proxy configuration around?  i was wonderin if it would be posible to extend kolla to allow ha proxy to be used only for internal traffic and disable it for the external traffic?15:21
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla: Fix l3_agent.ini rendering error when DVR  https://review.openstack.org/38191915:22
pbourkebjolo: i have a fix for your neutron issue15:22
pbourkebjolo: can you give it a go?15:22
pbourkebjolo: https://review.openstack.org/38191915:23
*** dave-mccowan has joined #openstack-kolla15:30
*** dwalsh has quit IRC15:32
sean-k-mooneyby the way when is the stable/newton branch planned to be made?15:33
*** eaguilar has quit IRC15:34
inc0kfox1111, sbezverk_ so nfs has issues, I think I'll try gluster instead15:35
rhalliseyinc0, what issues are you seeing?15:36
rhalliseyinc0, if we have nothing else, could go with ceph and have a script15:37
inc0mariadb has issues with file locks15:37
inc0rhallisey, but we don't have means to run this script15:37
rhalliseydid it break your ceph?15:37
inc0I haven't try;)15:38
inc0what sbezverk_ was saying makes sense15:38
inc0we need some way to run fencing script15:38
*** salv-orlando has joined #openstack-kolla15:40
kfox1111rhallisey: or just use the kubectl evacuate to simulate node failure. it will do the right thing.15:41
*** srwilkers has quit IRC15:41
kfox1111yes. I've not tried it with kolla-k8s, but have seen the issue first hand on another k8s system.15:41
kfox1111if you kill the node, it wont release the rbd lock on volumes mounted.15:42
kfox1111its kind of hard to say if its a ceph issue or a k8s one.15:42
inc0I think it's k8s issue and ceph safety15:44
*** mnasiadka has quit IRC15:45
kfox1111all rbd clients have the issue though. not just ceph.15:45
kfox1111all rbd clients have the issue though. not just k8s.15:45
kfox1111k8s just has enough extra knowledge to maybe work around it if ceph doesnt.15:46
inc0gluster would give us both ha and might not suffer from same issue15:46
* kfox1111 shrugs.15:46
kfox1111I think cephfs might work too.15:46
kfox1111the issue is block based with locks on the dedicated volume.15:47
*** matrohon has quit IRC15:47
inc0I'm pretty sure we can manually release lock on rbd, but what we would need is some logic that k8s could run on pod evac15:47
kfox1111shared file systems don't have that issue.15:47
kfox1111but they perform worse then dedicated block device.15:47
kfox1111so kind of a tradeoff.15:47
kfox1111yeah, manual lock evacuation works. I've trid it.15:48
inc0issue is, how to release it?15:48
kfox1111it really belongs in the code that probes kubelet aliveness and adds fencing too, as the node could still be alive.15:48
kfox1111that parts firly simple. the rbd command has an unlock command.15:48
*** dwalsh has joined #openstack-kolla15:48
kfox1111for the demo, I think we just write a "killnode.sh" script, that does:15:48
inc0what I meant what will release it15:49
kfox11111, ipmi power off, 2, drops the lock.15:49
inc0(I'm sick hence my english is failing:()15:49
*** huikang has joined #openstack-kolla15:49
kfox1111rbd lock remove imagename <lock-id> <locker>15:49
kfox1111you get lock-id and locker from rbd lock list15:49
inc0http://docs.ceph.com/docs/jewel/man/8/rbd/ --shared lock-tag15:50
inc0how about this?15:50
kfox1111k8s won't issue it without patching.15:51
openstackgerritWaldemar Znoinski (wznoinsk) proposed openstack/kolla: WIP: setting basic structures for iscsi container  https://review.openstack.org/38194215:51
inc0really? we can't specify flags of ceph rbd?15:52
kfox1111nope.15:52
kfox1111normally you don't need to.15:52
*** daneyon has joined #openstack-kolla15:52
kfox1111and I don't think even rbd is doing it...15:52
kfox1111I think the kernel issues the lock.15:52
kfox1111I just think we should wait this one out. come up with something demo'able, but k8s will get a work around eventually.15:54
kfox1111as is though, we only have 2 volumes, and I think we really could do with just one.15:54
kfox1111and after we get mariadb clustering, even the lock issue won't be an issue, as there will be other members that will survive a node failure.15:55
inc0what will happen if we issue lock remove before anything happends?15:56
inc0right after rbd is created?15:56
*** daneyon has quit IRC15:57
kfox1111the lock only gets created when its mounted.,15:57
kfox1111so when kubelet launches the pod, it rbd map voumename, to /dev/rbd0, which takes the lock,15:58
kfox1111then it gets mounted into /var/lib/kubelet/volumes....15:58
kfox1111then the node dies, and nothing's there to pull the lock.15:58
kfox1111so the next pod fails while trying to rbd map volumename15:58
inc0but if rbd is mounted15:58
*** dwalsh has quit IRC15:58
inc0we remove lock, rbd keeps being mounted15:59
inc0after restart lock is already removed so we should be able to just remount15:59
kfox1111yeah. so pulling a lock should only be done when you know for sure a node is dead.15:59
kfox1111the issue is:15:59
kfox1111 node dies, it had  the lock, it isn't there to release the lock.15:59
inc0you sure? what if node is alive and well, aka we pull lock right after initial mount15:59
kfox1111another node tries to launch the pod, tries to take the lock, its still there, it waits forever for the lock to release that won't ever get released.16:00
inc0ok, let me just try what I'm thinking might be the issue;)16:00
kfox1111all you have to do in that situation is delete the lock. k8s will recover it then.16:00
*** lamt has joined #openstack-kolla16:01
*** mark-casey1 has joined #openstack-kolla16:03
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs  https://review.openstack.org/38104116:03
kfox1111so for the demo, the script to kill the node just has to release the lock too. everything shoudl work then.16:04
*** huikang has quit IRC16:06
inc0k8s uses ssh to create mons?16:06
*** huikang has joined #openstack-kolla16:06
kfox1111not sure I understand the question.16:07
*** diogogmt_ has joined #openstack-kolla16:07
kfox1111s/k8s/kolla-kubernetes/; s/mons/volumes/ ?16:08
inc0yes...16:08
kfox1111ah.16:08
inc0sorry, sick, language failing.16:08
kfox1111no worries. :)16:08
kfox1111that part is still in development.16:08
*** diogogmt has quit IRC16:08
*** diogogmt_ is now known as diogogmt16:08
kfox1111the ps I'm working on literally right now: https://review.openstack.org/38104116:08
kfox1111does it a different way.16:08
kfox1111I make an ceph-admin container, and kubectl exec stuff into it.16:09
kfox1111much more k8s'ish.16:09
inc0also when I specify ceph key...I base64 key that is base64?16:10
kfox1111rhallisey: https://review.openstack.org/#/c/380848/ <- landed. you should have a hook now to start testing the ansible bits.16:10
kfox1111k8s expects secrets to be base64'ed. it doesn't care what it is, it base64 decodes it on the other side.16:11
*** dwalsh has joined #openstack-kolla16:11
kfox1111ceph expects a base64 encoded key too.16:11
rhalliseykfox1111, the main gate was suppose to run those bits16:11
kfox1111so it needs to be double base64 encoded.16:11
*** huikang has quit IRC16:11
kfox1111rhallisey: I don't follow. what main gate?16:12
rhalliseyyou added a second job template16:13
kfox1111rhallisey: all it gives an extra flag now to the setup_get.sh. so we can branch there and fire off ansible instead of the shell script bits.16:13
kfox1111yeah. it calls the same job in the end. it just had to be a different template because the args were a little different.16:14
kfox1111and I didn't want to permutate all possible options.16:14
kfox1111so its only 5 jobs, instead of 8.16:15
rhalliseythat's fine16:16
rhalliseywfm16:17
rhalliseygood way to test the anisble bits anyway16:17
kfox1111yeah. and being merged in to the script that way, we can just branch around the parts we have in ansible, and continue to use shell for the rest,16:18
kfox1111that way we can replace a piece at a time and still validate everything deploys ok.16:18
sean-k-mooneyhi out of interest is there anything happening at the summit on the monday16:20
sean-k-mooneyis there an operator day like austin?16:20
*** david-lyle has joined #openstack-kolla16:21
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: Start testing the deployments  https://review.openstack.org/38086816:22
kfox1111sbezverk_: I think the keepalived issue was maybe that br-ex's interface is down... not sure anything brought it up.16:23
kfox1111goint to switch back to the external-ip resource to see if it fares any better.16:23
kfox1111http://logs.openstack.org/68/380868/21/experimental/gate-kolla-kubernetes-deploy-centos-binary-ceph-nv/aaa7983/logs/ip.txt you can see br-ex is in state down and has no ip. but in the rest of the logs keepalived's running and seems happy.16:24
*** openstackgerrit has quit IRC16:26
*** openstackgerrit has joined #openstack-kolla16:27
*** Pavo has quit IRC16:27
*** openstackgerrit has quit IRC16:28
*** openstackgerrit has joined #openstack-kolla16:28
*** openstackgerrit has quit IRC16:30
*** openstackgerrit has joined #openstack-kolla16:30
*** Pavo has joined #openstack-kolla16:33
sbezverk_kfox1111: makesure network_interface is defined16:33
kfox1111I explicitly defined api_interface.16:34
kfox1111bbiab16:35
inc0sbezverk_, how can I debug pv creation?16:38
sbezverk_inc0: any errors messages?16:41
inc0I can't see any, but at the same time I don't know where to look:)16:41
inc0so I created pv, it's there16:41
inc0do rbd ls in ceph - > nothing16:41
inc0mariadb bootstrap hangs on SuccessfulCreate Created pod16:42
sbezverk_inc0: kubel get pvc --namesapce=kolla16:42
sbezverk_inc0: kubel get pv --namesapce=kolla16:42
sbezverk_on compute node where pod got started check /var/log/messages16:43
sbezverk_mount | grep rbd16:43
sbezverk_lsmod | grerp rbd16:43
sbezverk_kubectl describe pod {mariadb pod name} --namespace=kolla16:44
inc0hmm, let see16:45
*** fguillot has quit IRC16:45
inc0hmm..yeah installation of ceph clients makes mounting rbd easier16:46
sbezverk_inc0: :-) you bet16:47
*** cj has joined #openstack-kolla16:51
cjhey folks16:51
cjdo we have a docker-compose setup available?16:51
SamYapleinc0: just fyi, kernel rbd mounting is not the best idea and doesnt work for voluems created in openstack due to the enabled features. best to stick with userspace16:51
inc0SamYaple, not sure how k8s deals with it16:52
inc0it might be userspace16:52
inc0but still kernel module seems useful;)16:52
*** eaguilar has joined #openstack-kolla16:53
inc0cj, docker compose?:)16:53
inc0we were compose based like...2 years ago16:53
kfox1111inc0: pv creation is just a record in k8s.16:54
SamYapleinc0: sure, but its old and outdated to the point that it actually _doesnt_ work for default rbd configuration in jewel.16:54
cjinc0: okay, that's no longer the new hotness? :-)16:54
SamYaplekernel only supports RBD_FEATURE_LAYERING | RBD_FEATURE_STRIPINGV216:54
kfox1111you have to create/format the rbd volumes yourself.16:54
kfox1111SamYaple: I think k8s onlyu supports kernel rbd at the moment.16:54
SamYaplekfox1111: ouch. sucks to be k8s16:55
SamYaplethats missing out on like all the good stuff16:55
inc0cj, didn't really meet our requirements, openstack doesn't handle this things well, we moved on to ansible16:55
*** msimonin1 has quit IRC16:55
cjokay.  I guess I'll look at ansible16:55
SamYaplehttps://github.com/torvalds/linux/blob/master/drivers/block/rbd.c#L119-12216:56
kfox1111SamYaple: yeah. It'll catch up eventually.16:56
*** huikang has joined #openstack-kolla16:56
SamYaplekfox1111: i know of no one working on the kernel module, but for what its worth, i did get it confirmed to me that the kernel module is _deprecated_ as the ML would have you believe16:56
inc0kfox1111, you're saying that I need to create rbd myself?16:56
kfox1111inc0: yes.16:57
inc0meh16:57
kfox1111inc0: there's the start of instructions here: https://review.openstack.org/#/c/381041/16:57
kfox1111meeting next. bbiab.16:57
*** strigazi is now known as strigazi_AFK17:00
*** vhosakot has joined #openstack-kolla17:03
inc0kfox1111, ok, I wasn't aware of that, then our lock issue is easy17:05
inc0just add --image-shared flag17:05
inc0to rbd create17:05
sbezverk_inc0: we have not tried that but I would assume other folks who hit this issue tried it17:06
inc0don't assume too much;)17:06
inc0when I get my ceph-backed stuff up I'll try17:06
sbezverk_inc0: please let me know if it helps, will rebuild my volumes17:07
*** eaguilar has quit IRC17:09
*** strigazi_AFK is now known as strigazi17:11
*** vhosakot has quit IRC17:15
*** vhosakot has joined #openstack-kolla17:15
*** strigazi is now known as strigazi_AFK17:22
inc0sbezverk_, kfox1111 look at this https://github.com/kubernetes/kubernetes/blob/3a42ec0b1df55c5ce96fe4e5c5c1b9fb0037f2c0/pkg/volume/rbd/rbd_util.go#L19917:24
rhalliseyinteresting17:26
*** vhosakot_ has joined #openstack-kolla17:27
*** tonanhngo has quit IRC17:29
*** vhosakot has quit IRC17:30
*** HyperJohnGraham has quit IRC17:37
*** eaguilar has joined #openstack-kolla17:47
*** fguillot has joined #openstack-kolla17:49
openstackgerritRyan Hallisey proposed openstack/kolla-kubernetes: Ansible support for Mariadb  https://review.openstack.org/33425517:51
openstackgerritRyan Hallisey proposed openstack/kolla-kubernetes: Ansible support for Rabbitmq  https://review.openstack.org/38079617:51
openstackgerritRyan Hallisey proposed openstack/kolla-kubernetes: Ansible workflow for Keystone  https://review.openstack.org/38079717:51
openstackgerritRyan Hallisey proposed openstack/kolla-kubernetes: Ansible workflow for Glance  https://review.openstack.org/38080117:51
openstackgerritRyan Hallisey proposed openstack/kolla-kubernetes: Workflow Gate Testing  https://review.openstack.org/38202117:51
*** inc0 has quit IRC17:52
*** bjolo_ has joined #openstack-kolla17:53
*** Serlex has quit IRC17:53
*** tonanhngo has joined #openstack-kolla17:56
*** sdake has quit IRC17:58
*** tonanhngo has quit IRC18:00
*** dwalsh has quit IRC18:01
*** ayoung has joined #openstack-kolla18:01
*** harlowja has quit IRC18:03
*** vinkman has joined #openstack-kolla18:03
*** lrensing has joined #openstack-kolla18:05
*** LamT__ has quit IRC18:11
*** vinkman has left #openstack-kolla18:12
*** tonanhngo has joined #openstack-kolla18:14
kfox1111back.18:14
*** tonanhngo has quit IRC18:14
kfox1111inc0, thats because k8s supports hving pods that share a read only volume.18:14
kfox1111that's a correct statement. no locking is used/needed when everyone shares a read only volume.18:15
kfox1111that doesn't help us though, as ours must be rw.18:15
*** dwalsh has joined #openstack-kolla18:16
*** eaguilar has quit IRC18:18
*** HyperJohnGraham has joined #openstack-kolla18:22
*** haplo37 has quit IRC18:25
*** athomas has quit IRC18:25
kfox1111uh.... weird...18:26
sbezverk_rhallisey: is there a way to test ansible workflows?18:26
kfox1111http://logs.openstack.org/68/380868/22/experimental/gate-kolla-kubernetes-deploy-centos-binary-ceph-nv/3780c69/console.html18:26
kfox1111seems like it got cut off half way through.18:27
kfox1111sbezverk_: rhallisey: yeah, we could start adding in the workflow bits into setup_gate.sh18:27
rhalliseysbezverk_, yes18:27
*** Pavo has quit IRC18:27
rhalliseyansible-playbook -i workflows/ansible/inventory/all-in-one workflows/ansible/site.yml -e @/etc/kolla-kubernetes/kolla-kubernetes.yml18:27
*** shardy is now known as shardy_afk18:27
sbezverk_rhallisey: oh nice, I was scratching my head on how to test it :-)18:28
*** Pavo has joined #openstack-kolla18:28
*** unicell1 has joined #openstack-kolla18:29
*** g3ek has quit IRC18:29
kfox1111whaaa?18:29
kfox1111http://logs.openstack.org/68/380868/22/experimental/gate-kolla-kubernetes-deploy-ubuntu-binary-ceph-nv/588148b/logs/openstack-catalog.txt18:29
kfox1111no public endpoint!?18:29
kfox1111arg....18:30
kfox1111we really need to break up our bootstrap jobs to 1 container, one thing. :/18:30
kfox1111the container seems to have failed, but told k8s it succeeded.18:30
*** g3ek has joined #openstack-kolla18:30
kfox1111yeah. the container is doing too much, it doesn't check for failure, and the last thing succeeded...18:32
sbezverk_kfox1111: we split already createdb/managedb/register endpoints18:33
*** senk_ has quit IRC18:33
sbezverk_kfox1111: maybe the issue when you batch things together?18:33
kfox1111no, I mean, split them to one job per task. create-endpoint-public, create-endpoint-internal, etc.18:33
kfox1111can use a template to share all the code though.18:33
kfox1111then if the one task fails, k8s will restart it.18:34
*** haplo37 has joined #openstack-kolla18:34
kfox1111so maybe we make a common-create-endpoint.yml.j2 template, that takes in a var of what service, and what endpoint.18:35
sbezverk_kfox1111: I noticed that restarting all tasks like createdb or maanagedb ot register endpoints usually causes bunch of troubles. as openstack does not handle well this operations of the same set of data18:35
sean-k-mooneyhas anyone else had issues with the rabbit mq contaienr constantly restarting on master? i and wznoinsk on our team have hit this18:36
kfox1111sbezverk_: you mean rerunning them after they have run successfully, or just letting them rerun after they fail?18:36
kfox1111splitting them up should help with the first case, as you only rerun the failed ones.18:36
*** ccesario has quit IRC18:38
*** ccesario has joined #openstack-kolla18:39
*** tonanhngo has joined #openstack-kolla18:44
*** tonanhngo_ has joined #openstack-kolla18:46
*** srwilkers has joined #openstack-kolla18:47
*** tonanhngo_ has quit IRC18:48
*** dwalsh has quit IRC18:48
*** salv-orlando has quit IRC18:48
*** tonanhngo has quit IRC18:48
*** tonanhngo_ has joined #openstack-kolla18:48
*** vhosakot_ has quit IRC18:48
sbezverk_kfox1111: I meant running then if they by any reason were rescheduled18:48
*** vhosakot has joined #openstack-kolla18:49
*** msimonin has joined #openstack-kolla18:50
sbezverk_kfox1111: do you want to take care of that?18:50
*** ssl_ has joined #openstack-kolla18:51
*** ssl_ is now known as ssl12318:51
kfox1111I can split them. should be too hard.18:51
ssl123i'm running a test and made changes to nova.conf, how do i restart the nova compute service within the nova-compute container?18:52
ssl123i can't do service nova-compute restart?18:52
*** v1k0d3n has quit IRC18:54
*** v1k0d3n has joined #openstack-kolla18:54
openstackgerritRyan Hallisey proposed openstack/kolla-kubernetes: Workflow Gate Testing  https://review.openstack.org/38202118:55
sean-k-mooneypbourke: mdnadeem  as an fyi i am hitting https://bugs.launchpad.net/kolla/+bug/1564773 on ubunut soucres with the head of  master also18:56
openstackLaunchpad bug 1564773 in kolla mitaka "Continously restarting rabbitmq container for CentOS" [High,Triaged] - Assigned to Md Nadeem (mail2nadeem92)18:56
*** lrensing has quit IRC18:56
kfox1111oh.... this is weird...18:58
kfox1111http://pastebin.com/XZTYdSrf18:58
sean-k-mooneykfox1111: you are seeing some of the endpoint multiple times ?18:59
*** dwalsh has joined #openstack-kolla18:59
kfox1111no, a little weirder. let me show you.19:00
kfox1111http://logs.openstack.org/68/380868/22/experimental/gate-kolla-kubernetes-deploy-ubuntu-binary-ceph-nv/588148b/logs/openstack-catalog.txt19:00
mark-casey1ssl123: I'm looking in the logs for a discussion I had that touches on some of that. The brief version is: long-term changes need to be made prior to deploy, short term testing can be done by changing the configs on the server/VM that runs the nova container in a particular place (because these get mounted into the container) and using docker to restart the container19:00
kfox1111I wonder if this is a race between parallel ansible jobs to build the endpoints.19:01
kfox1111each doesn't see the service_endpoint, so they create one of their own.19:01
*** ccesario has quit IRC19:01
kfox1111sometimes they run serially though, so things merge in that case.19:02
sean-k-mooneykfox1111: so is that why the glance image public enpoint is seperate form the admin and internal19:02
sean-k-mooneykfox1111: and why the cinderv2 service ans 3 seperate endpoints19:03
*** vhosakot has quit IRC19:03
kfox1111yeah. I think so..... going to test this theory...19:03
ssl123mark-casey1: i had some packages installed in the container.. if i restart the nova container, i assume i will lost those packages?19:03
*** vhosakot has joined #openstack-kolla19:03
sean-k-mooneykfox1111: where as keystone has a singel endpoint for interna public and admin19:03
ssl123is there a way for me just to restart the nova-compute service within the container?19:03
kfox1111say we have 2 endpoint creations about to go....19:03
kfox1111a and b.19:03
kfox1111a starts up, sees no glance service registred, so decides to register its own.19:04
sean-k-mooneyssl123: restarting the container should not effeect the filesystem so your packages should still exist19:04
mark-casey1ssl123: I don't think that's the way to do that but I'm not convinced those packages would go away19:04
kfox1111b starts up sees no glance service registerd, so decides to register its own.19:04
mark-casey1sean-k-mooney: right thanks :)19:04
kfox1111b registers a new glance endpoint, getting back uuid 1111-1111-1111-111119:04
kfox1111a registers a  new glance endpoint, getting back uuid 2222-222-222-222219:05
kfox1111a registers its url with that service uuid.19:05
kfox1111b registers its url with its service uuid.19:05
*** v1k0d3n has quit IRC19:05
kfox1111fun.19:05
mark-casey1ssl123: you can certainly get direct help, like just now, but as a fallback this irc log has me dealing with some of this same topic a few months back: http://eavesdrop.openstack.org/irclogs/%23openstack-kolla/%23openstack-kolla.2016-04-20.log.html19:07
sean-k-mooneyinteresting. kfox1111 i guess the registrations will have to be done serially then to prevent this.19:07
mark-casey1my question started "if I wanted to test a different value in nova.conf" and evolved into some discussion of how things work19:07
kfox1111sean-k-mooney: yeah... I think we can still parallize them, so long as there are only one of each service running. so 3 parallel sets. one admin run, one internal, and one public.19:08
*** v1k0d3n has joined #openstack-kolla19:08
sbezverk_kfox1111: they look independent, the only dependency if keystone up and running19:09
kfox1111yeah. except for the service id creation dependency.19:09
kfox1111the other way to do it would be a seperate job to create the service_id's.19:10
sean-k-mooneykfox1111: does the regeistration take place once per serverice per type(adming,internal,public) or is it also per contoler instance?19:10
kfox1111then all the endpoint creation could be in parallel.19:10
kfox1111sean-k-mooney: the former.19:10
sean-k-mooneyok well if its not liniarly dependent on the number of contolers then i dont think there will be a scaling reason to require a seperate step to precreate teh service_ids but it might be nicer to do that anyway19:11
kfox1111yeah.19:12
kfox1111that can be done in a followup ps though. maybe I just split and order in the first ps, and then we can do that in a followup.19:13
sean-k-mooneyok i have got to run. am before i do kfox1111 do you know when stable/newton will be cut in kolla?19:14
kfox1111I'm afraid I may need to just do this all in one big ps. :/19:14
kfox1111sean-k-mooney: I don't know. :/19:14
sean-k-mooneymaster seems to be kindof broken at the moment19:14
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: Start testing the deployments  https://review.openstack.org/38086819:16
sean-k-mooneyin the last week or 2 i have hit 3 or 4 seperate bugs with ceph,rabbitmq,heat and magnum(that is fixed)19:17
sean-k-mooneyanyway im an hour and a half late for going home so got to run19:18
*** sean-k-mooney is now known as sean-k-mooneyAFK19:18
sbezverk_rhallisey: we should proabably remove multinode inventory since it is not applicable to kube at all.19:19
kfox1111+119:20
rhallisey that's true19:20
rhalliseyreally only need one19:20
rhalliseysince you're not deploy on machiens19:20
kfox1111maybe just rename all-in-one 'kubernetes' or something?19:20
kfox1111so your running with the "kubernetes" inventory? :)19:21
sbezverk_or kube-api19:21
kfox1111that works too.19:22
sbezverk_since it is the entry point for us and api ip address would be there19:22
kfox1111kollakube? :)19:22
*** david-lyle has quit IRC19:22
kfox1111-i kollakube.... :)19:22
sbezverk_kollakube is good19:22
rhalliseyidc19:23
rhalliseyany of those19:23
kfox1111arg... like 3/4 of the time, the ceph tests fail in the ceph ps.19:23
kfox1111every time pg 1.1 is stuck...19:24
kfox1111its like ceph never fully deploys.19:24
kfox1111there may be a network config issue there...19:24
*** HyperJohnGraham has quit IRC19:26
sbezverk_kfox1111: hm but iscsi seems working ;-) maybe you should reconsider your view of it19:27
*** vhosakot has quit IRC19:27
*** egonzalez90 has joined #openstack-kolla19:27
*** egonzalez90 has quit IRC19:27
kfox1111sbezverk_: nope. when you can forklift an lvm based system with zero downtime, I'll re'evaluate it. ;)19:28
kfox1111the lvm backend is not ha.19:29
kfox1111it can be made ha, but its harder to get ha then ceph.19:29
*** daneyon has joined #openstack-kolla19:29
kfox1111I think this is more of a: the openstack test infrastructure is very heterogenious....19:30
kfox1111I think if I land on some nodes, it works, on others it doesnt.19:30
kfox1111I've gotten several reports that the minikube docs work fine, which is ceph based.19:31
kfox1111so its something different between minikube and the gate nodes.19:31
*** daneyon has quit IRC19:34
sbezverk_kfox1111: for sure, I remember some time ago when kolla PS hit a specific gate host it was always failing..19:34
kfox1111ouch.19:34
ssl123thanks mark-casey119:36
*** ssl123 has quit IRC19:37
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs  https://review.openstack.org/38104119:37
openstackgerritRyan Hallisey proposed openstack/kolla-kubernetes: Workflow Gate Testing  https://review.openstack.org/38202119:39
*** matrohon has joined #openstack-kolla19:40
*** shardy_afk is now known as shardy19:41
*** inc0 has joined #openstack-kolla19:41
inc0hello19:41
inc0apperantly I'm alergic to Texas.. I just came back from doctor and I have alergic reaction for first time in my life19:42
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: Start testing the deployments  https://review.openstack.org/38086819:43
rhalliseyinc0, ya I get that sometime when I go to different areas19:43
kfox1111rhallisey: any chance we can do the functionizing in a different ps? We've got 3 ps's now all working on setup_gate. merge conflicts with the non indented version are going to be a bit painful by themselves.19:44
kfox1111once all 3 are merged, then maybe we clean it all up?19:45
kfox1111and maybe even break it up into multiple files...19:45
rhalliseykfox1111, not need to merge mine for a bit19:45
rhalliseykfox1111, i need the functions since there are 2 ways to run this19:45
rhalliseyyou guys an merge first19:45
kfox1111k.19:45
rhalliseyall the workflows aren't done anyway19:46
rhalliseythis is easy wip19:46
rhalliseys/easy/early19:46
kfox1111well, maybe just after k8s is deployed, just call out to a fuction for ansible and then exit, ignoring the rest?19:46
kfox1111it could merge easier, and then we can functionize the rest.19:46
rhalliseyI'm fine with rebasing a few times19:47
rhalliseyit's no big deal19:47
kfox1111k.19:47
rhalliseyjust add where you guys need to19:47
kfox1111I haven't had rebases that big go well. but if you know how, great. :)19:47
rhalliseyI've had my practice19:47
rhallisey:)19:47
kfox1111cool.19:48
kfox1111yeah. every time I do it I learn something new. :)19:48
rhalliseygood stuff19:48
*** bjolo_ has quit IRC19:49
*** bjolo_ has joined #openstack-kolla19:49
inc0kfox1111, https://review.openstack.org/#/c/381041/27 correct me if I'm wrong, but if I have ceph client somewhere I don't really need it right?19:50
openstackgerritMauricio Lima proposed openstack/kolla: [WIP] Add HNAS as Manila backend  https://review.openstack.org/38205419:50
inc0this whole thing is just to have toolset to create stuff like pool or rbd image19:50
*** salv-orlando has joined #openstack-kolla19:51
sbezverk_inc0: if you have ceph-common install you do not need it19:51
kfox1111inc0: yeah. this ps is about smoothing out the deployment.19:52
HyperJohnGraham_sbezverk_: i'm at my desk now19:52
kfox1111so if you go and manuallyl create the rbd volumes elsewhere, or you preinstall ceph on all your k8s nodes, then you don't need it.19:52
sbezverk_HyperJohnGraham_: 2 more minutes, need to grab a tea19:53
kfox1111and its about adding gate checks for it all.19:53
HyperJohnGraham_ok cool19:53
kfox1111right now, the gate deploys mariadb/rabbitmq backed with hostpaths, which isn't ha.19:53
kfox1111this is trying to get it past the finish line where everything's out of ceph and ha.19:53
kfox1111meeting next. bbiab.19:54
*** salv-orlando has quit IRC19:55
sbezverk_HyperJohnGraham_: https://cisco.webex.com/join/sbezverk19:57
HyperJohnGraham_cool19:57
*** salv-orlando has joined #openstack-kolla19:59
*** salv-orlando has quit IRC19:59
*** salv-orlando has joined #openstack-kolla20:00
*** bjolo_ has quit IRC20:01
*** shardy has quit IRC20:03
*** david-lyle has joined #openstack-kolla20:07
openstackgerritRyan Hallisey proposed openstack/kolla-kubernetes: Workflow Gate Testing  https://review.openstack.org/38202120:12
*** lamt has quit IRC20:12
*** harlowja has joined #openstack-kolla20:15
*** HyperJohnGraham has joined #openstack-kolla20:16
*** david-lyle has quit IRC20:19
*** g3ek has quit IRC20:19
*** david-lyle has joined #openstack-kolla20:19
*** g3ek has joined #openstack-kolla20:20
*** Pavo has quit IRC20:27
*** lamt has joined #openstack-kolla20:29
*** Pavo has joined #openstack-kolla20:29
*** srwilkers has quit IRC20:30
*** srwilkers has joined #openstack-kolla20:30
*** coolsvap has quit IRC20:32
*** Pavo has quit IRC20:32
*** srwilkers has quit IRC20:33
*** Jeffrey4l_ has joined #openstack-kolla20:34
*** Jeffrey4l has quit IRC20:35
*** shardy has joined #openstack-kolla20:39
*** mark-casey1 has quit IRC20:41
*** v1k0d3n has quit IRC20:42
*** v1k0d3n has joined #openstack-kolla20:49
*** diogogmt_ has joined #openstack-kolla20:59
sbezverk_inc0: ping20:59
inc0I'm here20:59
sbezverk_inc0: from the top of your head, why do you think fetching ceph key task from ceph cluster might fail?21:00
sbezverk_btw it is kolla, not kolla-kube21:00
inc0ssh issue?21:01
*** diogogmt has quit IRC21:01
*** diogogmt_ is now known as diogogmt21:01
inc0remember that it requires nodes to ssh to each other21:01
inc0can21:01
inc0can't remember which ones;)21:01
sbezverk_inc0: everything else heka/toolsbox got deployed though21:02
inc0so fetching keyrings is delegated task21:02
inc0https://github.com/openstack/kolla/blob/master/ansible/roles/ceph/tasks/distribute_keyrings.yml#L721:03
*** david-lyle has quit IRC21:04
sbezverk_inc0: I see21:04
sbezverk_inc0: does it matter if it is the same machine?21:05
inc0you have 1 mon?21:05
sbezverk_inc0: yes21:06
inc0hmm21:07
inc0then it's probably not this, what is full error?21:07
*** huikang has quit IRC21:08
*** huikang has joined #openstack-kolla21:09
sbezverk_inc0: error is coming from John21:09
HyperJohnGraham_http://pastebin.com/p0Xx1dxJ21:09
inc0https://github.com/openstack/kolla/blob/99897c5438f59b3fa40ade388e0eafe6a0fbfffb/doc/ceph-guide.rst#deploy-fails-with-fetching-ceph-keyrings--no-json-object-could-be-decoded21:11
inc0:)21:11
sbezverk_inc0: it seems ceph_mon is not running that is why it fails , but I cannot see why starting ceph_mon was skipped21:12
inc0^21:12
inc0docker volume ls21:13
openstackgerritRyan Hallisey proposed openstack/kolla-kubernetes: Workflow Gate Testing  https://review.openstack.org/38202121:13
HyperJohnGraham_http://pastebin.com/zgTCShQK21:13
inc0HyperJohnGraham_, did you click the link I pasted?:)21:13
*** huikang has quit IRC21:13
HyperJohnGraham_http://paste.openstack.org/show/584314/21:14
inc0HyperJohnGraham_, remove both ceph volumes before redeploying21:14
inc0this error happends when you have stale volumes21:15
sbezverk_inc0: thank you will try it21:15
*** shardy has quit IRC21:15
*** matrohon has quit IRC21:15
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: Start testing the deployments  https://review.openstack.org/38086821:17
kfox1111sbezverk_: have a look at this one when you get a chance... just the common jobs part. If I can get it all passing the gate checks, I might try and break it out to a seperate ps. it was just convenient to test here, as it has a lot more debuging in place.21:18
*** ppalacios has quit IRC21:18
*** vhosakot has joined #openstack-kolla21:19
*** senk_ has joined #openstack-kolla21:20
HyperJohnGraham_http://paste.openstack.org/show/584317/21:22
*** schwicht has quit IRC21:24
*** eaguilar has joined #openstack-kolla21:25
sbezverk_inc0: it went further but not too much, could you check again?21:25
sbezverk_kfox1111: the only concern is  -e "{'service_auth':{{ global[serviceAuth] }}}"21:27
sbezverk_does it get rendered ok?21:27
kfox1111k. the gate check will verify that works or not.21:27
kfox1111I believe it will.21:27
inc0HyperJohnGraham_, when you see "container exited with non-zero code" next step should always be docker logs container-name21:28
inc0so docker logs ceph_mon21:28
sbezverk_inc0: we checked that  and there was no any obivous cause of crash21:28
inc0what is non-obvious one?:) logs plz21:29
*** jheroux has quit IRC21:30
sbezverk_inc0: 2 seconds John stepped out21:30
*** msimonin has quit IRC21:30
sbezverk_inc0: basically what we do it to use kolla-ansible to install ceph cluster and then to use it on John's kubernetes cluster which we just installed and configured21:31
openstackgerritChristian Berendt proposed openstack/kolla: Create /var/log/kolla/rally before running rally-manage db create/upgrade  https://review.openstack.org/38207421:32
inc0can I join in to webex? I try to do something similar21:32
inc0I have ceph installed, I have k8s installed, but could use pointers on how to merge these two21:32
sbezverk_absolutely!!!!21:32
HyperJohnGraham_http://paste.openstack.org/show/584318/21:33
sbezverk_inc0: here is the link21:33
sbezverk_https://cisco.webex.com/join/sbezverk21:33
inc0guys, you need to remove these volumes21:33
*** david-lyle has joined #openstack-kolla21:33
*** vhosakot_ has joined #openstack-kolla21:34
*** vhosakot has quit IRC21:35
*** huikang has joined #openstack-kolla21:36
openstackgerritMerged openstack/kolla: Fix l3_agent.ini rendering error when DVR  https://review.openstack.org/38191921:37
*** senk_ has quit IRC21:38
*** MarMat has quit IRC21:39
*** MarMat has joined #openstack-kolla21:41
*** david-lyle has quit IRC21:42
*** vhosakot_ has quit IRC21:42
MarMatanyone testing magnum currently? It seems broken to me due to that change in magnum https://github.com/openstack/magnum/commit/324f4aca7d194991a028068a9b8acef6f74e11c921:42
openstackgerritMerged openstack/kolla: Fix sahara endpoint url  https://review.openstack.org/38118821:44
*** diogogmt has quit IRC21:44
openstackgerritRyan Hallisey proposed openstack/kolla-kubernetes: Workflow Gate Testing  https://review.openstack.org/38202121:44
*** david-lyle has joined #openstack-kolla21:47
*** david-lyle has quit IRC21:48
*** david-lyle has joined #openstack-kolla21:48
inc0HyperJohnGraham_, https://github.com/openstack/kolla/blob/master/doc/production-architecture-guide.rst#docker-configuration21:48
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: Start testing the deployments  https://review.openstack.org/38086821:49
*** schwicht has joined #openstack-kolla21:50
*** dwalsh has quit IRC21:53
*** fguillot has quit IRC21:58
*** senk_ has joined #openstack-kolla22:01
*** eaguilar_ has joined #openstack-kolla22:02
*** eaguilar has quit IRC22:03
*** schwicht has quit IRC22:04
openstackgerritRyan Hallisey proposed openstack/kolla-kubernetes: Workflow Gate Testing  https://review.openstack.org/38202122:05
*** lamt has quit IRC22:08
*** dave-mccowan has quit IRC22:11
*** daneyon has joined #openstack-kolla22:11
sbezverk_inc0: ceph auth get-key client.admin22:13
sbezverk_AQBAMo1VqE1OMhAAVpERPcyQU5pzU6IOJ22x1w==22:13
sbezverk_echo "AQBAMo1VqE1OMhAAVpERPcyQU5pzU6IOJ22x1w==" | base6422:13
sbezverk_QVFCQU1vMVZxRTFPTWhBQVZwRVJQY3lRVTVwelU2SU9KMjJ4MXc9PQo=22:13
sbezverk_Edit your ceph-secret.yml with the base64 key:22:13
sbezverk_apiVersion: v122:13
sbezverk_kind: Secret22:13
sbezverk_metadata:22:13
sbezverk_  name: ceph-secret22:13
sbezverk_data:22:13
sbezverk_  key: QVFCQU1vMVZxRTFPTWhBQVZwRVJQY3lRVTVwelU2SU9KMjJ4MXc9PQo=22:13
sbezverk_kubectl create -f secret/ceph-secret.yaml22:13
*** daneyon has quit IRC22:16
*** huikang has quit IRC22:17
*** huikang has joined #openstack-kolla22:18
*** schwicht has joined #openstack-kolla22:20
*** huikang has quit IRC22:22
sbezverk_inc0: http://paste.openstack.org/show/584324/22:24
*** huikang has joined #openstack-kolla22:25
*** huikang has quit IRC22:30
*** schwicht has quit IRC22:39
kfox1111nice admin key. :)22:40
*** senk_ has quit IRC22:40
inc0it's 256 of 2016 already?22:41
inc00 256 016.M2 as warhammer 40k people would describe it22:44
kfox1111:)22:45
kfox1111you play?22:45
*** eaguilar_ has quit IRC22:46
*** eaguilar has joined #openstack-kolla22:48
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: Start testing the deployments  https://review.openstack.org/38086822:49
*** schwicht has joined #openstack-kolla22:49
kfox1111lets see if the broken out/common bootstrap jobs fixes the issue. :)22:49
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs  https://review.openstack.org/38104122:53
*** rhallisey has quit IRC22:55
*** senk_ has joined #openstack-kolla22:55
inc0kfox1111, pen and paper, video games and tons of books22:58
*** sdake has joined #openstack-kolla22:58
inc0basically my favorite sf univere22:58
*** salv-orlando has quit IRC23:01
kfox1111yeah. I really like 40k. just haven't had much time for it lately. :(23:03
*** sdake_ has joined #openstack-kolla23:05
*** diogogmt has joined #openstack-kolla23:06
*** sdake has quit IRC23:09
*** diogogmt has quit IRC23:09
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: Start testing the deployments  https://review.openstack.org/38086823:09
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs  https://review.openstack.org/38104123:13
*** senk_ has quit IRC23:17
*** HyperJohnGraham has quit IRC23:18
*** rodrigo_pereira has quit IRC23:21
*** schwicht has quit IRC23:22
kfox1111nice... the idea does seem to have wings. :)23:26
kfox1111it fixed the multiple service id's thing for sure. :)23:26
kfox1111still running... will see if all the endpoints get created now.23:26
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs  https://review.openstack.org/38104123:33
sdake_hey peeeps23:38
sdake_sorry internet is down23:38
sdake_at parents atm23:38
sdake_not sure what is busted23:38
sdake_anyone blocked or need help?23:38
kfox1111http://logs.openstack.org/68/380868/28/experimental/gate-kolla-kubernetes-deploy-centos-binary-ceph-nv/54cfea3/console.html almost there... so close... :)23:39
kfox1111glance image uploaded, networks created, vm's starting to start... cli commands changed. :/23:40
*** senk_ has joined #openstack-kolla23:42
*** eaguilar has quit IRC23:46
*** awiddersheim has quit IRC23:47
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: Start testing the deployments  https://review.openstack.org/38086823:49
*** awiddersheim has joined #openstack-kolla23:53
kfox1111theres something just really weird about ceph in the gate...23:53
*** haplo37_ has quit IRC23:58

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!