| sdake | sbezverk_ i can tell yo uwhy | 00:00 |
|---|---|---|
| sdake | networking performance | 00:00 |
| sdake | docker proxy is slower then a 28.8k modem | 00:00 |
| sbezverk_ | docker proxy is still running in docker and since they changed proxy to use iptables I was told that the performance imrpoved | 00:01 |
| sbezverk_ | I meant kube proxy | 00:01 |
| sdake | i see | 00:01 |
| sdake | maybe i'm wrong | 00:01 |
| sdake | been drinking - has happened before :) | 00:01 |
| *** vinkman has quit IRC | 00:03 | |
| *** tonanhngo has joined #openstack-kolla | 00:04 | |
| *** tonanhngo has quit IRC | 00:05 | |
| *** tonanhngo has joined #openstack-kolla | 00:07 | |
| *** tonanhngo has quit IRC | 00:08 | |
| *** tonanhngo has joined #openstack-kolla | 00:08 | |
| *** HyperJohnGraham has joined #openstack-kolla | 00:18 | |
| *** diogogmt has joined #openstack-kolla | 00:20 | |
| sbezverk_ | kfox1111: if you need I can test part of your doc, kube integration with just ceph cluster without bells and whistels | 00:24 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 00:24 |
| kfox1111 | sbezverk_: k. | 00:25 |
| kfox1111 | that would be cool. gota head out very soon though. been up since 1:30am this morning. :/ | 00:25 |
| sbezverk_ | kfox1111: ok I will reply to PS if I hit anything | 00:27 |
| kfox1111 | k. thx. | 00:28 |
| *** msimonin has joined #openstack-kolla | 00:28 | |
| kfox1111 | its such an odd thing. both ps's seem like they should be working, or really close... but things are just sketchy... | 00:28 |
| kfox1111 | very similar stuff in minkube just works. | 00:28 |
| *** root____8 has joined #openstack-kolla | 00:29 | |
| *** root____8 has left #openstack-kolla | 00:30 | |
| *** slivi323 has joined #openstack-kolla | 00:30 | |
| *** daneyon has joined #openstack-kolla | 00:32 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: Start testing the deployments https://review.openstack.org/380868 | 00:34 |
| kfox1111 | there. I officially give up on it being ready soon. :/ | 00:35 |
| sdake | rbergeron around? | 00:36 |
| sdake | kfox1111 gate changes hard - don't get discouraged :) | 00:36 |
| *** daneyon has quit IRC | 00:36 | |
| sdake | kfox1111 if its too much to tkae on in one patch split it up into multiple chunks of work | 00:37 |
| sdake | thats what i typically do | 00:37 |
| sdake | to try to make the work less | 00:37 |
| kfox1111 | I'm fine. just overly tired at this point. | 00:37 |
| kfox1111 | I'm too stubbern to let the gate win. ;) | 00:37 |
| kfox1111 | well.... there's a rax node... | 00:39 |
| kfox1111 | and its takign forever... | 00:39 |
| britthouser2 | I want a t-shirt that says, "I'm too stubbern to let the gate win." | 00:39 |
| sdake | britthouser2 that would be winning | 00:40 |
| sdake | we ought to put that on the cisco shirts ;) | 00:40 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: Adding kubectl commands to setup default namespace https://review.openstack.org/381349 | 00:40 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 00:41 |
| kfox1111 | now I'm just making stupid mistakes. :/ | 00:41 |
| kfox1111 | k. have a good one folks. I'm going to bed. :) | 00:42 |
| britthouser2 | Gnight kfox1111. Sleep to fight the gate another day. | 00:42 |
| sbezverk_ | kfox1111: take care | 00:42 |
| kfox1111 | you too. | 00:44 |
| HyperJohnGraham | hi all | 00:45 |
| *** slivi323 has quit IRC | 00:54 | |
| *** MarMat has quit IRC | 00:54 | |
| *** jax3242 has quit IRC | 00:55 | |
| *** phuongnh has joined #openstack-kolla | 00:59 | |
| *** v1k0d3n has joined #openstack-kolla | 01:00 | |
| *** awidders_ has quit IRC | 01:03 | |
| *** awiddersheim has joined #openstack-kolla | 01:03 | |
| *** zhubingbing has joined #openstack-kolla | 01:05 | |
| *** zhubingbing has quit IRC | 01:18 | |
| *** zhubingbing has joined #openstack-kolla | 01:19 | |
| *** duonghq has joined #openstack-kolla | 01:29 | |
| *** msimonin has quit IRC | 01:31 | |
| *** zhubingbing has quit IRC | 01:35 | |
| sbezverk_ | HyperJohnGraham: finally rebuild my cluster and got mariadb running so tomorrow I can work with you to get yours going | 01:38 |
| HyperJohnGraham | oh very cool ! | 01:38 |
| HyperJohnGraham | i have meetings in the morning but should be free by noon pacific time | 01:40 |
| *** schwicht_at_work has joined #openstack-kolla | 01:42 | |
| sbezverk_ | ok it is not the best time for me because people tend to break networks around 3PM EST ;-) but hopefully tomorrow it will not be the case | 01:42 |
| *** zhubingbing has joined #openstack-kolla | 01:42 | |
| *** zhubingbing has quit IRC | 01:43 | |
| *** schwicht has quit IRC | 01:43 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Add hexdump package for ironic-conductor container https://review.openstack.org/381238 | 01:44 |
| *** zhubingbing has joined #openstack-kolla | 01:47 | |
| *** zhubingbing has quit IRC | 01:52 | |
| *** inc0 has quit IRC | 01:59 | |
| sdake | can someone let pbourke know that jewel needs to be implemented in oracle linux | 02:04 |
| *** MarMat has joined #openstack-kolla | 02:05 | |
| *** unicell has quit IRC | 02:07 | |
| sdake | marmat how is magnum looking | 02:12 |
| HyperJohnGraham | sbezverk_: early morning too | 02:19 |
| HyperJohnGraham | but i have a 10am dentist appointment i need to break for | 02:20 |
| HyperJohnGraham | ping me in the am any time if you want | 02:21 |
| sdake | sbezverk_ my finest work is done at 3pm est ;) | 02:22 |
| *** david-lyle has joined #openstack-kolla | 02:25 | |
| *** Pavo has quit IRC | 02:27 | |
| *** tonanhngo has quit IRC | 02:28 | |
| MarMat | sdake looks fine with me... you mean anything particular? | 02:31 |
| *** tonanhngo has joined #openstack-kolla | 02:31 | |
| *** Pavo has joined #openstack-kolla | 02:33 | |
| *** david-lyle has quit IRC | 02:38 | |
| *** david-lyle has joined #openstack-kolla | 02:49 | |
| *** david-lyle has quit IRC | 02:55 | |
| *** caowei has joined #openstack-kolla | 03:00 | |
| *** g3ek has quit IRC | 03:00 | |
| *** caowei has quit IRC | 03:01 | |
| *** haplo37 has quit IRC | 03:01 | |
| *** g3ek has joined #openstack-kolla | 03:07 | |
| *** haplo37 has joined #openstack-kolla | 03:07 | |
| *** unicell has joined #openstack-kolla | 03:13 | |
| *** daneyon has joined #openstack-kolla | 03:14 | |
| *** daneyon has quit IRC | 03:19 | |
| *** eaguilar has quit IRC | 03:21 | |
| sdake | marmat is it good to ship | 03:31 |
| MarMat | sdake subjectivey so far ok, still running tests on newest builds and different configurations... are there any criteria to be "good to ship"? | 03:35 |
| sdake | marmat a) does it work is criteria 1 | 03:37 |
| sdake | b) does it work well is criteria 2 | 03:37 |
| sdake | if those ren't met, then we releasenote it as techpreview | 03:38 |
| sdake | at ummit we will come up with a better mechanism for handling what we consider the inner core, outer core, and extra outer core | 03:38 |
| sdake | or whatever | 03:38 |
| sdake | who knows :) | 03:38 |
| sdake | actually if a is met, we dont need a techpreviewe releasenote | 03:39 |
| MarMat | sdake well basic configuration (no barbican, no lb, cinder lvm) is working. More complex configurations either need some workarounds or we have not tested yet. | 03:42 |
| sdake | would be curious to know if cinder ceph works | 03:42 |
| sdake | that would be good enough for me | 03:42 |
| MarMat | sdake there are still some weeks for decision, right? | 03:43 |
| sdake | 8 days | 03:43 |
| MarMat | sdake ask me then in 8 days :-) | 03:43 |
| MarMat | sdake is anyone else testing magnum? | 03:44 |
| sdake | i used to | 03:44 |
| sdake | but no time now | 03:44 |
| sdake | so your our go to on that decision :) | 03:44 |
| sdake | note its the cinese national holiday so we are missing some of our ch contribs | 03:45 |
| sdake | i think for a week :( | 03:45 |
| sdake | Jeffrey4l__ still seems to be at it | 03:45 |
| sdake | but iiuc he is our highest output contributor | 03:45 |
| Jeffrey4l__ | sup | 03:45 |
| sdake | Jeffrey4l__ not a ping just mentioning your name | 03:46 |
| Jeffrey4l__ | Got ;) | 03:46 |
| MarMat | sdake roger that, will proceed with some more magnum tests. | 03:46 |
| sdake | cinder + ceph would be good | 03:47 |
| HyperJohnGraham_ | :) | 03:47 |
| sdake | its too late this cycle to set quality standards for components | 03:47 |
| sdake | we know our core is fantastic | 03:47 |
| sdake | and the outer core (things like heat/horizon/etc) work fantastic | 03:47 |
| sdake | the rest of the added services - unknown | 03:48 |
| MarMat | sdake talking about quality, a question - how it came that the rabbit bump issues from Friday were not detected by CI/checks/gates? | 03:50 |
| sdake | they were | 03:50 |
| sdake | the gate was down fri-mon | 03:50 |
| sdake | early am my time | 03:50 |
| sdake | but our ci checks are weak | 03:51 |
| sdake | they need alot of attention | 03:51 |
| MarMat | I see | 03:51 |
| *** yuanying has quit IRC | 04:01 | |
| *** yuanying_ has joined #openstack-kolla | 04:01 | |
| *** jmccarthy has quit IRC | 04:03 | |
| *** jmccarthy has joined #openstack-kolla | 04:04 | |
| openstackgerrit | Merged openstack/kolla: Revert "Fix pycparser wheel package issue" https://review.openstack.org/381237 | 04:06 |
| openstackgerrit | Merged openstack/kolla: Add hexdump package for ironic-conductor container https://review.openstack.org/381238 | 04:07 |
| openstackgerrit | Merged openstack/kolla: install pxelinux for ironic_pxe https://review.openstack.org/380350 | 04:07 |
| *** absubram has joined #openstack-kolla | 04:17 | |
| *** absubram_ has joined #openstack-kolla | 04:18 | |
| *** absubram has quit IRC | 04:21 | |
| *** absubram_ is now known as absubram | 04:21 | |
| *** Pavo has quit IRC | 04:27 | |
| *** salv-orlando has joined #openstack-kolla | 04:29 | |
| *** lamt has quit IRC | 04:31 | |
| *** Pavo has joined #openstack-kolla | 04:32 | |
| *** salv-orlando has quit IRC | 04:34 | |
| *** senk__ has joined #openstack-kolla | 04:39 | |
| *** senk__ has quit IRC | 04:39 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla: Updated from global requirements https://review.openstack.org/375989 | 05:03 |
| *** unicell1 has joined #openstack-kolla | 05:04 | |
| *** unicell has quit IRC | 05:05 | |
| *** salv-orlando has joined #openstack-kolla | 05:07 | |
| *** MarMat has quit IRC | 05:12 | |
| bjolo_ | morning | 05:21 |
| bjolo_ | sdake, still up? ok to build or still issues? | 05:24 |
| *** unicell1 has quit IRC | 05:29 | |
| *** unicell has joined #openstack-kolla | 05:30 | |
| *** g3ek has quit IRC | 05:34 | |
| *** haplo37 has quit IRC | 05:34 | |
| *** g3ek has joined #openstack-kolla | 05:39 | |
| *** msimonin has joined #openstack-kolla | 05:40 | |
| *** haplo37 has joined #openstack-kolla | 05:40 | |
| *** flaper87 has joined #openstack-kolla | 05:49 | |
| *** flaper87 has quit IRC | 05:49 | |
| *** flaper87 has joined #openstack-kolla | 05:49 | |
| *** daneyon has joined #openstack-kolla | 05:57 | |
| *** daneyon has quit IRC | 06:01 | |
| *** Jeffrey4l__ has quit IRC | 06:14 | |
| *** MartinsG has quit IRC | 06:14 | |
| *** Jeffrey4l__ has joined #openstack-kolla | 06:14 | |
| openstackgerrit | gecong proposed openstack/kolla-kubernetes: Replace LOG.warn with LOG.warning https://review.openstack.org/381429 | 06:14 |
| *** Jeffrey4l__ has quit IRC | 06:14 | |
| *** Jeffrey4l_ has joined #openstack-kolla | 06:15 | |
| *** Jeffrey4l_ is now known as Jeffrey4l | 06:15 | |
| *** salv-orlando has quit IRC | 06:21 | |
| *** msimonin has quit IRC | 06:24 | |
| *** Pavo has quit IRC | 06:27 | |
| openstackgerrit | Takashi Sogabe proposed openstack/kolla-kubernetes: A fix of tunnel_interface in neutron-openvswitch-agent https://review.openstack.org/381434 | 06:29 |
| *** Pavo has joined #openstack-kolla | 06:34 | |
| sdake | bjolo build is good to go | 06:36 |
| sdake | bjolo yup still up just got home from folks house | 06:36 |
| sdake | bjolo headed to bed tho - night | 06:36 |
| jmccarthy | Hi all ! | 06:39 |
| jmccarthy | Ceph question if anyone knows, why is fstab updated with info to mount disks on the hosts ? | 06:39 |
| jmccarthy | Aren't these mounted directly by the containers for themselves anyway ? | 06:40 |
| *** egonzalez90 has joined #openstack-kolla | 06:40 | |
| *** tonanhngo has quit IRC | 06:46 | |
| *** unicell1 has joined #openstack-kolla | 06:50 | |
| *** unicell has quit IRC | 06:51 | |
| *** tonanhngo has joined #openstack-kolla | 06:53 | |
| *** hieulq has joined #openstack-kolla | 06:54 | |
| *** tonanhngo has quit IRC | 06:54 | |
| openstackgerrit | Merged openstack/kolla: Change mysql-check in HAProxy to post MySQL 4.1 https://review.openstack.org/381218 | 07:01 |
| *** senk_ has joined #openstack-kolla | 07:04 | |
| *** senk_ has quit IRC | 07:06 | |
| *** ankush has joined #openstack-kolla | 07:09 | |
| *** salv-orlando has joined #openstack-kolla | 07:11 | |
| duonghq | anybody see zhubingbing around? | 07:14 |
| *** matrohon has joined #openstack-kolla | 07:14 | |
| *** msimonin has joined #openstack-kolla | 07:14 | |
| bjolo_ | jmccarthy, dont have kolla ceph deployed myself. Does the containers mount /etc/fstab from the host? | 07:16 |
| *** tonanhngo has joined #openstack-kolla | 07:16 | |
| *** tonanhngo has quit IRC | 07:19 | |
| jmccarthy | bjolo_ I'd have to look, I'm not that familiar with the implementation actually, I'm think of swift (which does not modify fstab and seems to work just fine) | 07:24 |
| *** b_bezak has joined #openstack-kolla | 07:27 | |
| *** salv-orlando has quit IRC | 07:33 | |
| *** salv-orlando has joined #openstack-kolla | 07:46 | |
| *** hieulq has quit IRC | 07:47 | |
| *** Serlex has joined #openstack-kolla | 07:51 | |
| duonghq | sdake: around? | 07:51 |
| *** g3ek has quit IRC | 07:58 | |
| *** haplo37 has quit IRC | 07:59 | |
| *** imcsk8 has quit IRC | 08:01 | |
| *** imcsk8 has joined #openstack-kolla | 08:01 | |
| sdake | duonghq yess just responding to an email | 08:01 |
| sdake | and ready to rack | 08:01 |
| sdake | whats up | 08:01 |
| duonghq | sdake: for the precheck task | 08:02 |
| duonghq | currently, we check through ping | 08:02 |
| duonghq | so, we don't cope with the blocked ICMP | 08:03 |
| duonghq | or we haven't coped with it | 08:03 |
| *** hieulq has joined #openstack-kolla | 08:03 | |
| *** g3ek has joined #openstack-kolla | 08:05 | |
| duonghq | (I mean the kolla_internal_vip_address check) | 08:05 |
| *** haplo37 has joined #openstack-kolla | 08:06 | |
| *** salv-orlando has quit IRC | 08:12 | |
| sdake | right | 08:13 |
| duonghq | sdake: so, when we cannot ping the ip, can we assume the IP does not existed | 08:15 |
| *** athomas has joined #openstack-kolla | 08:15 | |
| *** salv-orlando has joined #openstack-kolla | 08:19 | |
| *** salv-orlando has quit IRC | 08:21 | |
| *** salv-orlando has joined #openstack-kolla | 08:21 | |
| sdake | duonghq right | 08:26 |
| sdake | duonghq have proposal to fix it? | 08:26 |
| sdake | duonghq i think its such a low priority issue at this point not worth disturbing the release over | 08:26 |
| sdake | but please file a bug | 08:26 |
| sdake | and make a case :) | 08:26 |
| *** msimonin has quit IRC | 08:27 | |
| *** Pavo has quit IRC | 08:27 | |
| *** hieulq has quit IRC | 08:28 | |
| duonghq | sdake: roger | 08:30 |
| *** msimonin has joined #openstack-kolla | 08:31 | |
| *** Pavo has joined #openstack-kolla | 08:32 | |
| *** hieulq has joined #openstack-kolla | 08:33 | |
| duonghq | sdake: it's really a small bug but I heard that in many case, org block icmp from switch/router manner? | 08:34 |
| *** tonanhngo has joined #openstack-kolla | 08:35 | |
| *** tonanhngo has quit IRC | 08:36 | |
| sdake | rbergeron you around | 08:39 |
| sdake | duonghq - would like to hear a suggesteddworkaround | 08:40 |
| sdake | if you can give me a 45 second ptich that would rock | 08:40 |
| sdake | else i am off to bed | 08:40 |
| duonghq | sdake: I'll check ssh port | 08:40 |
| duonghq | because Ansible work through ssh | 08:40 |
| duonghq | so check if the ssh open is quite resonable? | 08:41 |
| sdake | works for me | 08:41 |
| duonghq | thank sdake, so now you hit the bed? | 08:42 |
| sdake | better then ping imo | 08:42 |
| sdake | although ansible ping usess ssh :) | 08:42 |
| sdake | maybe we are using ping directly which is a problem | 08:42 |
| duonghq | sdake: ya, I'll rewrite that, | 08:42 |
| duonghq | I'm always wonder why we use raw ping | 08:43 |
| duonghq | so I'll switch to ping module | 08:43 |
| duonghq | hope that it's good enough | 08:43 |
| *** quasar has joined #openstack-kolla | 08:49 | |
| *** tonanhngo has joined #openstack-kolla | 08:54 | |
| *** tonanhngo has quit IRC | 08:57 | |
| sdake | rbegeron ping | 09:02 |
| sdake | duonghq ya ping module uses ssh pretty sure | 09:02 |
| duonghq | sdake: the ping module is not designed to use in playbook | 09:03 |
| duonghq | so wait_for is under duty | 09:03 |
| *** strigazi_AFK is now known as strigazi | 09:04 | |
| sdake | hmm i can't say i understand why | 09:04 |
| sdake | (the ping module in playbooks) | 09:04 |
| sdake | anyway - time to crash sorry dude ;) | 09:05 |
| duonghq | okay sdake, see you tommorrow | 09:05 |
| *** mnasiadka has joined #openstack-kolla | 09:10 | |
| *** gfidente has joined #openstack-kolla | 09:11 | |
| *** jrich523 has quit IRC | 09:23 | |
| *** jrich523 has joined #openstack-kolla | 09:26 | |
| coolsvap | duonghq: can you point me to the review in discussion? | 09:32 |
| *** daneyon has joined #openstack-kolla | 09:33 | |
| duonghq | coolsvap: I'm working on the ps, but the bug is https://bugs.launchpad.net/kolla/+bug/1630165 | 09:34 |
| openstack | Launchpad bug 1630165 in kolla "Improve VIP existence check" [Wishlist,New] - Assigned to Duong Ha-Quang (duonghq) | 09:34 |
| duonghq | coolsvap: it's quite interesting case, indeed | 09:34 |
| coolsvap | duonghq: so what you intend to use? | 09:34 |
| duonghq | I intend use wait_for module | 09:35 |
| duonghq | I check for ssh_port open | 09:36 |
| coolsvap | wait_for is fine, so do you have a test system where you can test the exact scenario you depicted in bug? | 09:36 |
| *** narasimha_SV has joined #openstack-kolla | 09:37 | |
| coolsvap | I have seen wait_for suffers from regression while checking for ssh | 09:37 |
| duonghq | sure, it's quite easy to create testbed | 09:37 |
| coolsvap | have seen that earlier, but we can always test it again | 09:37 |
| narasimha_SV | http://paste.openstack.org/show/584146/ while dpeloyiing keystone contianer is failing is with this error | 09:37 |
| duonghq | but the problem is I cannot determine with port is used for ssh in the VIP | 09:37 |
| duonghq | *with -> which | 09:38 |
| *** daneyon has quit IRC | 09:38 | |
| coolsvap | i suppose you can start with default and improvise for different ports in next iteration | 09:43 |
| duonghq | coolsvap: sure, so I'll try with 22, due to there is no clue for other port | 09:43 |
| *** shardy has joined #openstack-kolla | 09:47 | |
| coolsvap | duonghq: sure | 09:50 |
| *** gfidente has quit IRC | 09:51 | |
| *** gfidente has joined #openstack-kolla | 09:53 | |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla: Improve VIP existence check https://review.openstack.org/381589 | 10:00 |
| *** zigo has quit IRC | 10:01 | |
| *** zigo has joined #openstack-kolla | 10:04 | |
| *** zigo is now known as Guest84780 | 10:05 | |
| *** senk_ has joined #openstack-kolla | 10:11 | |
| *** senk_ has quit IRC | 10:12 | |
| bjolo_ | hi all | 10:14 |
| *** msimonin has quit IRC | 10:14 | |
| bjolo_ | question about reviews and merged code | 10:14 |
| bjolo_ | im waiting for this PS https://review.openstack.org/#/c/373455/ | 10:14 |
| *** Guest84780 has quit IRC | 10:14 | |
| bjolo_ | it says merged, but i dont see the code when i do a git clone | 10:14 |
| pbourke | bjolo_: that change is merged | 10:14 |
| *** senk_ has joined #openstack-kolla | 10:15 | |
| pbourke | https://github.com/openstack/kolla/commit/d1673ad173ea96c0e018946f54caa99776855cc1 | 10:16 |
| bjolo_ | ah i need to update globals myself | 10:16 |
| bjolo_ | the start.yml code is there | 10:16 |
| *** zigo_ has joined #openstack-kolla | 10:16 | |
| bjolo_ | nevermind my tired head :) | 10:17 |
| *** msimonin has joined #openstack-kolla | 10:17 | |
| *** msimonin has quit IRC | 10:17 | |
| *** senk_ has quit IRC | 10:21 | |
| *** Pavo has quit IRC | 10:27 | |
| *** duonghq has quit IRC | 10:27 | |
| *** g3ek has quit IRC | 10:28 | |
| *** haplo37 has quit IRC | 10:28 | |
| *** ntpttr has quit IRC | 10:30 | |
| *** srwilkers has quit IRC | 10:30 | |
| *** ntpttr has joined #openstack-kolla | 10:31 | |
| *** Pavo has joined #openstack-kolla | 10:32 | |
| *** g3ek has joined #openstack-kolla | 10:34 | |
| *** phuongnh has quit IRC | 10:35 | |
| *** haplo37 has joined #openstack-kolla | 10:35 | |
| *** salv-orlando has quit IRC | 10:42 | |
| *** coolsvap is now known as coolsvap_ | 11:01 | |
| *** senk_ has joined #openstack-kolla | 11:10 | |
| *** bjolo_ is now known as bjolo | 11:10 | |
| bjolo | question about horizon and policy.json files | 11:15 |
| *** rhallisey has joined #openstack-kolla | 11:15 | |
| bjolo | my understanding is that horizon needs to have the same policy file as keystone has | 11:16 |
| bjolo | but today there is no copy config for policy files for horizon | 11:16 |
| bjolo | would that be deemed a bug and fixed in newton, or considered new feature and moved to ocata? | 11:17 |
| *** zigo_ is now known as zigo | 11:23 | |
| *** haplo37_ has quit IRC | 11:33 | |
| pbourke | bjolo: depends what the result is having them the same | 11:35 |
| *** haplo37_ has joined #openstack-kolla | 11:35 | |
| *** narasimha_SV has quit IRC | 11:39 | |
| *** salv-orlando has joined #openstack-kolla | 11:41 | |
| openstackgerrit | Merged openstack/kolla-kubernetes: A fix of tunnel_interface in neutron-openvswitch-agent https://review.openstack.org/381434 | 11:44 |
| *** dwalsh has joined #openstack-kolla | 11:45 | |
| bjolo | pbourke, in my use case im trying to get keystone policy.v3cloudsample.json working | 11:46 |
| *** salv-orlando has quit IRC | 11:46 | |
| bjolo | not suceeded all the way yet, but one part is to make horizon work with domains, and for that i need to copy the policy file into the container. no biggi to do manually but i would be nicer to have kolla doing it | 11:48 |
| openstackgerrit | Merged openstack/kolla-kubernetes: Replace LOG.warn with LOG.warning https://review.openstack.org/381429 | 11:48 |
| bjolo | can any core reviewer please do a triage on this one? https://bugs.launchpad.net/kolla/+bug/1626995 | 11:50 |
| openstack | Launchpad bug 1626995 in kolla "neutron_dvr fails to deploy" [Undecided,New] | 11:51 |
| pbourke | bjolo: hmm I think I've seen this | 11:53 |
| bjolo | should it be removed or completed? | 11:55 |
| bjolo | seems like the code is kinda cut-off | 11:55 |
| pbourke | can you post the deploy output with -vvv | 11:56 |
| bjolo | ok | 11:56 |
| *** ppalacios has joined #openstack-kolla | 12:00 | |
| *** tonanhngo has joined #openstack-kolla | 12:01 | |
| *** msimonin has joined #openstack-kolla | 12:01 | |
| *** tonanhngo has quit IRC | 12:02 | |
| *** rodrigo_pereira has joined #openstack-kolla | 12:02 | |
| ppalacios | Am I missing something or images 2.0.2 are not properly configured? I'm trying to "kolla-ansible deploy" and when I reach the Horizon playbook, the task "creating the member role" fails after 10 tries. It looks like that the ansible within this image are missing extra modules. Does anybody can help? | 12:12 |
| pbourke | ppalacios: please pastebin the full log preferably with -vvv | 12:13 |
| *** zhubingbing has joined #openstack-kolla | 12:15 | |
| *** daneyon has joined #openstack-kolla | 12:16 | |
| *** srwilkers has joined #openstack-kolla | 12:16 | |
| *** portdirect has quit IRC | 12:20 | |
| ppalacios | pbourke: just a sec (I'm running it again) | 12:20 |
| *** daneyon has quit IRC | 12:21 | |
| *** Pavo has quit IRC | 12:27 | |
| *** schwicht_at_work has quit IRC | 12:28 | |
| *** salv-orlando has joined #openstack-kolla | 12:28 | |
| openstackgerrit | Matthieu Simonin proposed openstack/kolla: Allow custom Type value in heka decoders. https://review.openstack.org/381764 | 12:32 |
| *** Pavo has joined #openstack-kolla | 12:32 | |
| *** tonanhngo has joined #openstack-kolla | 12:34 | |
| *** tonanhngo has quit IRC | 12:36 | |
| *** fguillot has joined #openstack-kolla | 12:37 | |
| *** msimonin1 has joined #openstack-kolla | 12:38 | |
| *** msimonin has quit IRC | 12:40 | |
| *** senk_ has quit IRC | 12:42 | |
| bjolo | pbourke, sorry taking a while. i had ironic enabled as well, and it seems ironic fails to deploy also | 12:47 |
| bjolo | pbourke, http://paste.openstack.org/show/584192/ | 12:56 |
| *** shardy is now known as shardy_mtg | 12:56 | |
| bjolo | the bug is updated as well with the log paste | 12:56 |
| pbourke | thanks bjolo looking into it now | 13:00 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: Fixing TUNNEL_INTERFACE in neutron dhcp agent pod https://review.openstack.org/381785 | 13:00 |
| kfox1111 | monring. | 13:02 |
| bjolo | good afternoon kfox1111 | 13:02 |
| kfox1111 | :) | 13:02 |
| bjolo | :) | 13:02 |
| britthouser2 | Hey pbourke - last night sdake said something about asking you about oracle linux and jewel (ceph version) | 13:02 |
| britthouser2 | did you get the message already? | 13:03 |
| bjolo | is etcd needed for kuryr? | 13:03 |
| bjolo | in globals.yml | 13:03 |
| bjolo | kolla-ansible | 13:04 |
| pbourke | britthouser2: no | 13:05 |
| pbourke | britthouser2: may have been around an issue myself and a few others were seeing in ceph recently | 13:05 |
| *** fguillot has quit IRC | 13:08 | |
| rhallisey | kfox1111, hey :) | 13:09 |
| *** dave-mccowan has joined #openstack-kolla | 13:11 | |
| ppalacios | pbourke: I got it solved here. I was trying to deploy with images from https://hub.docker.com/u/kolla/ . After building the images by myself I was able to complete all playbooks. So, the remaning question is, should I always build the images by myself? | 13:11 |
| pbourke | ppalacios: yes it's generally the best idea. currently we dont have a good mechanism of keeping the dockerhub builds up to date | 13:11 |
| kfox1111 | rhallisey: hey. :) | 13:13 |
| sbezverk_ | good morning | 13:14 |
| kfox1111 | sbezverk_: good monring. :) | 13:14 |
| ppalacios | pbourke: thanks! | 13:15 |
| sbezverk_ | kfox1111: ceph-rbd purpose is just to have ceph-common and distribute ceph keyring, correct? | 13:15 |
| *** schwicht has joined #openstack-kolla | 13:15 | |
| kfox1111 | sbezverk_: no. its a way not to have to install ceph on the hosts. | 13:15 |
| sbezverk_ | kfox1111: that is what I meant :-) | 13:16 |
| kfox1111 | it distributes a payload of a /usr/bin/rbd script that runs rbd in docker. | 13:16 |
| *** egonzalez90 has quit IRC | 13:16 | |
| kfox1111 | sthe ceph conf and key still come through normally. | 13:16 |
| kfox1111 | I did tweak the instructions though so that you deploy the ceph key via secret though. | 13:17 |
| britthouser2 | I tried to find it in my scrollback pbourke, but not seeing it right off. | 13:17 |
| sbezverk_ | kfox1111: I was looking into that, since we already have secret, why not to use it.. | 13:18 |
| kfox1111 | I'm working on getting rid of the admin key. its dangerous. | 13:19 |
| kfox1111 | so the ceph-acmin container and key can be completely deleted from kubernetes once the volumes/users are inited. | 13:19 |
| kfox1111 | ceph-admin | 13:20 |
| *** tonanhngo has joined #openstack-kolla | 13:20 | |
| *** tonanhngo has quit IRC | 13:20 | |
| kfox1111 | the gate check sets up a kolla ceph user for the rbd volumes. | 13:20 |
| pbourke | britthouser2: no worries thanks anyway | 13:20 |
| bjolo | sdake, are you up? | 13:21 |
| sbezverk_ | kfox1111: it sounds a little as a overcomplication at this stage, but that does not stop you ;-) | 13:22 |
| kfox1111 | sbezverk_: As always, I'm interested in prduction worthyness. :) | 13:22 |
| bjolo | can i please get triage on this one https://bugs.launchpad.net/kolla/+bug/1626958 | 13:22 |
| openstack | Launchpad bug 1626958 in kolla "neutron_server not working when lbaas enabled" [Undecided,New] | 13:22 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 13:23 |
| kfox1111 | almost there I think... just gota figure out the automation for formating the volumes. | 13:23 |
| pbourke | bjolo: still on it | 13:23 |
| sbezverk_ | kfox1111: btw without --image-feature layering I could not attach from kube to ceph volume, so I think it is a mandatory.. | 13:24 |
| kfox1111 | sbezverk_: could be. this is hammer though. that doesn't work with hammer. | 13:24 |
| kfox1111 | once we get a 3.0.0 set of containers released, I'll add a conditional in there for that. | 13:25 |
| *** ccesario has joined #openstack-kolla | 13:26 | |
| sbezverk_ | kfox1111: ok | 13:26 |
| sbezverk_ | rhallisey: I added kubectl commands to quick start so you do not have to type --namespace=kolla, could you please test it in your env? | 13:27 |
| rhallisey | ok cool | 13:27 |
| sbezverk_ | rhallisey: https://review.openstack.org/381349 | 13:28 |
| kfox1111 | sbezverk_: keepalived seems to not quite be working right: http://logs.openstack.org/68/380868/21/experimental/gate-kolla-kubernetes-deploy-centos-binary-ceph-nv/aaa7983/console.html | 13:29 |
| kfox1111 | curling the cluster ip works, but the external ip times out. | 13:29 |
| sbezverk_ | kfox1111: hm, I have it running in multinode and it works great. | 13:31 |
| kfox1111 | yeah. probably just misconfigured? | 13:32 |
| *** schwicht has quit IRC | 13:33 | |
| sbezverk_ | kfox1111: can you add ping of VIP after keepalived starts? | 13:34 |
| kfox1111 | http://logs.openstack.org/68/380868/21/experimental/gate-kolla-kubernetes-deploy-centos-binary-ceph-nv/aaa7983/logs/ip.txt | 13:34 |
| kfox1111 | doesn't apear to be attached. | 13:34 |
| *** janem has joined #openstack-kolla | 13:34 | |
| sbezverk_ | also getting logs from keepalived container would be helpfull | 13:34 |
| kfox1111 | not much in the logs: http://logs.openstack.org/68/380868/21/experimental/gate-kolla-kubernetes-deploy-centos-binary-ceph-nv/aaa7983/logs/pods/kolla-keepalived-xbcm7-main.txt | 13:35 |
| *** tonanhngo has joined #openstack-kolla | 13:36 | |
| *** tonanhngo has quit IRC | 13:37 | |
| *** ccesario has quit IRC | 13:37 | |
| *** inc0 has joined #openstack-kolla | 13:38 | |
| sbezverk_ | kfox1111: can you check and paste configmap generated? | 13:40 |
| *** skramaja has quit IRC | 13:41 | |
| kfox1111 | one of the few things I'm not capturing yet. :/ will have to do another run. | 13:42 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: Adding kubectl commands to setup default namespace https://review.openstack.org/381349 | 13:47 |
| rhallisey | sdake, hey | 13:47 |
| rhallisey | or inc0 | 13:47 |
| rhallisey | can you define trivialfix | 13:47 |
| *** schwicht has joined #openstack-kolla | 13:47 | |
| inc0 | I don't think there is definition | 13:48 |
| rhallisey | can you define when to use it | 13:48 |
| rhallisey | this has confused me | 13:48 |
| rhallisey | is it purely for docs? Or is it meant for things that are trivial fixes | 13:48 |
| kfox1111 | rhallisey: "I'll know it when I see it" ? :) | 13:49 |
| inc0 | no, it's not meant only for docs | 13:49 |
| inc0 | it's meant for one liners which are obvious and so onm | 13:49 |
| rhallisey | ok that's what I see it as too | 13:49 |
| kfox1111 | k | 13:50 |
| inc0 | so you won't need to publish bug to every small improvement | 13:50 |
| inc0 | lower cost of improvement = more improvements | 13:50 |
| rhallisey | kfox1111, reason I'm asking is beacuse it's sort of an indicator that subsitutes bugs/bps | 13:50 |
| *** zhubingbing has quit IRC | 13:50 | |
| rhallisey | just want to be sure we're tracking the best way | 13:50 |
| inc0 | rhallisey, we should slowly limit it's ussage in kolla, but kolla-k8s can go ahead | 13:50 |
| inc0 | it's a tradeoff - with it it's harder to put change into context | 13:51 |
| *** sean-k-mooneyAFK is now known as sean-k-mooney | 13:51 | |
| inc0 | which is important long run when core team will stop handling following every workstream in community | 13:51 |
| inc0 | which is kinda happening now:) | 13:51 |
| wznoinsk | hi all | 13:54 |
| rhallisey | inc0, agreed | 13:54 |
| *** senk_ has joined #openstack-kolla | 13:54 | |
| kfox1111 | yeah | 13:54 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla: Updated from global requirements https://review.openstack.org/375989 | 13:54 |
| wznoinsk | would anyone have some experience with iscsid container? asking because I'm planning to use it as dependency for ironic-conductor instead of having iscsid in conductor's container - I can't see any testing of this hence would need somebody to eyeball my proposal | 13:55 |
| *** egonzalez90 has joined #openstack-kolla | 13:56 | |
| *** ccesario has joined #openstack-kolla | 13:57 | |
| *** dwalsh has quit IRC | 13:57 | |
| *** MarMat has joined #openstack-kolla | 13:58 | |
| *** dwalsh has joined #openstack-kolla | 13:58 | |
| *** diogogmt has quit IRC | 13:59 | |
| *** coolsvap_ is now known as coolsvap | 13:59 | |
| *** dwalsh has quit IRC | 14:00 | |
| pbourke | inc0: hey inc0, just wondering did you ever get round to that ceph too many pgs bug? | 14:00 |
| pbourke | inc0: if not I can look at it | 14:00 |
| -openstackstatus- NOTICE: The Gerrit service on review.openstack.org is being restarted to address performance degradation and should return momentarily | 14:01 | |
| inc0 | pbourke, could you? I really need to get myself into kolla-k8s;) | 14:01 |
| *** dwalsh has joined #openstack-kolla | 14:01 | |
| pbourke | inc0: np | 14:01 |
| inc0 | I'm to present on summit about it after all;) | 14:01 |
| inc0 | let me know if you need any help | 14:01 |
| inc0 | also Sean's mons are working | 14:01 |
| *** pbourke has quit IRC | 14:02 | |
| *** salv-orlando has quit IRC | 14:02 | |
| *** pbourke has joined #openstack-kolla | 14:03 | |
| *** tonanhngo has joined #openstack-kolla | 14:03 | |
| *** eaguilar has joined #openstack-kolla | 14:03 | |
| *** senk_ has quit IRC | 14:03 | |
| *** tonanhngo has quit IRC | 14:03 | |
| sean-k-mooney | pbourke: i have hit that too in the past i think the main issue is that the the amount of pgs are hardcoded. it would be great if that was a config option. | 14:04 |
| pbourke | sean-k-mooney: yeah it needs to be fixed. Issue is last time I tried ceph master Im still getting the mon error | 14:05 |
| sean-k-mooney | pbourke: i was able to get past it by not using 6 mons. i was goint to look at the pg bug once i finished getting this cluster working but its probably a week out before i will have time | 14:06 |
| *** shardy_mtg is now known as shardy | 14:06 | |
| *** matrohon has quit IRC | 14:06 | |
| pbourke | sean-k-mooney: I use 3 mons | 14:06 |
| *** dwalsh has quit IRC | 14:06 | |
| sean-k-mooney | using 3 mons and removing dumb-init worked for me | 14:07 |
| sean-k-mooney | i havent tested with dumb-init | 14:07 |
| sbezverk_ | inc0: if you have not put together kube multinode, I suggest you check new kubeadm, with minor parameters change, I brought up 5 node cluster and kolla-kube runs on it.. | 14:07 |
| inc0 | sbezverk_, I've already done it:) | 14:07 |
| HyperJohnGraham | morning all | 14:08 |
| inc0 | https://github.com/inc0/kargo | 14:08 |
| sean-k-mooney | pbourke: i also have get a fully working cluster as heat was failing so i havent tried spawning vms but i got passed the mons hanging. | 14:08 |
| pbourke | inc0: sean-k-mooney: rhallisey: anyone opposed to installing the clients in kolla-toolbox? | 14:08 |
| inc0 | pbourke, nah, but why? | 14:08 |
| pbourke | inc0: handy as a toolbox | 14:08 |
| pbourke | inc0: given its name :) | 14:09 |
| inc0 | you'd also need to put admin.sh there | 14:09 |
| pbourke | i have a script that uses it just haven't had time to submit it yet | 14:09 |
| pbourke | passing in the vars etc. | 14:09 |
| pbourke | maybe will wait and submit both together | 14:09 |
| rhallisey | pbourke,kolla-k8s is doing it | 14:10 |
| pbourke | oh? | 14:10 |
| rhallisey | pbourke, they use to be in there | 14:10 |
| rhallisey | there were deps of shade | 14:10 |
| pbourke | yeah it has openstackclient | 14:10 |
| *** jheroux has joined #openstack-kolla | 14:11 | |
| pbourke | but its missing some that arent deps | 14:11 |
| rhallisey | ya I'd say add them | 14:11 |
| sean-k-mooney | rhallisey: oh sepaking of shade i hit a sade error with heat yesterday. | 14:11 |
| rhallisey | really | 14:11 |
| rhallisey | bug? | 14:11 |
| sean-k-mooney | im rebuilding my images to see if its still hapening on master but it was complaining that it needed shade version 1.5 or newer | 14:12 |
| sean-k-mooney | sorry i have a seperate issue with heat the shade issue was with magnum | 14:12 |
| *** gfidente has quit IRC | 14:13 | |
| sean-k-mooney | the tasks that setup the magnum trustee was failing. ill see if i can recreate and ill open a bug if i can | 14:13 |
| *** fguillot has joined #openstack-kolla | 14:14 | |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla: Install openstack clients in kolla-toolbox https://review.openstack.org/381876 | 14:15 |
| rhallisey | sean-k-mooney, ok gotcha | 14:16 |
| sean-k-mooney | by the way is it expect that i should be able to run a deploy after runing destroy when using ceph? do i need to relable the partitions for the osd to be bootstraped again? | 14:17 |
| *** dwalsh has joined #openstack-kolla | 14:20 | |
| *** matrohon has joined #openstack-kolla | 14:21 | |
| *** MartinsG has joined #openstack-kolla | 14:22 | |
| *** lrensing has joined #openstack-kolla | 14:23 | |
| pbourke | sean-k-mooney: I find they need to be relabelled | 14:23 |
| pbourke | sean-k-mooney: in fact yeah that is the case | 14:24 |
| pbourke | sean-k-mooney: destroy should prob do that | 14:24 |
| pbourke | sean-k-mooney: they need to be unmounted also I have a patch up for that | 14:24 |
| MartinsG | Hi, guys! I was wondering if there are any plans on implementing SELinux policies for qemu. Without Docker libvirt has SELinux policies, which allow running each qemu-kvm in separate selinux domain and secure it. | 14:24 |
| pbourke | MartinsG: none currently though I think we have a meeting this week with someone from the security group | 14:25 |
| pbourke | MartinsG: so I can mention it if you like, or feel free to join and bring it up :) | 14:25 |
| sean-k-mooney | pbourke: what i have seen in the past is that it deploys the osd's but they stay restarting if i dont relable. | 14:25 |
| sean-k-mooney | pbourke: having destroy read be able to restory the original lables would be useful. | 14:26 |
| *** ccesario has quit IRC | 14:27 | |
| *** Pavo has quit IRC | 14:27 | |
| *** ccesario has joined #openstack-kolla | 14:28 | |
| *** denaitre has joined #openstack-kolla | 14:28 | |
| MartinsG | pbourke: Thank you! :) | 14:29 |
| *** b_bezak has quit IRC | 14:29 | |
| pbourke | sdake: inc0: was it thursday for the security meeting? | 14:30 |
| inc0 | yeah | 14:30 |
| pbourke | inc0: time? | 14:31 |
| pbourke | (will send a ML) | 14:31 |
| inc0 | 3pm utc | 14:31 |
| *** Pavo has joined #openstack-kolla | 14:32 | |
| sbezverk_ | rhallisey: ping | 14:33 |
| *** tonanhngo has joined #openstack-kolla | 14:35 | |
| MarMat | sean-k-mooney what is exactly the error with shade? I have bumped shade to 1.5.0 in kolla-toolbox because of magnum | 14:35 |
| *** tonanhngo has quit IRC | 14:36 | |
| rhallisey | sbezverk_, yes sir | 14:36 |
| MarMat | sean-k-mooney magnum is building fine for (for centos) | 14:37 |
| rhallisey | sbezverk_, I'm not opposed to your patch | 14:37 |
| rhallisey | btw | 14:37 |
| *** tonanhngo has joined #openstack-kolla | 14:37 | |
| rhallisey | I think though, there is more to be done there | 14:37 |
| sbezverk_ | rhallisey: that part I get :-) | 14:37 |
| sbezverk_ | it is just the idea of using user's current context for the namespace for kolla services does not sound right to me.. | 14:38 |
| sbezverk_ | but I might be wrong :-) | 14:39 |
| rhallisey | where I'd think it's ok to have 2 sources of truth here is if the op will ignore these steps because they want to stay in the deafult namespace | 14:40 |
| sbezverk_ | rhallisey: fine in this case they will configure kolla_namespace: default and that is it | 14:41 |
| rhallisey | the reason for the error checking is because it's a common mistake to not know your namespace | 14:41 |
| *** senk_ has joined #openstack-kolla | 14:42 | |
| *** bmace has quit IRC | 14:42 | |
| rhallisey | sbezverk_, I'm ok with merging as is and leaving kolla_namespace var in place. I think we can revisit precheck/logging for this though | 14:43 |
| sbezverk_ | rhallisey: idk I am afraid the error checking logic could be very complex. how do you propose detection of wrong namespace in cli? | 14:43 |
| rhallisey | check current namespace and compare to config file namespace | 14:44 |
| sean-k-mooney | MarMat: it was building fine but when kolla tried to deploy it failed. i belive it was failing here https://github.com/openstack/kolla/blob/869aa073b549c837b32a8538a40f98d419ce5a84/ansible/roles/magnum/tasks/register.yml#L60-L73 | 14:44 |
| rhallisey | not equal just throw a one time warning | 14:44 |
| sbezverk_ | rhallisey: ok got it.. | 14:45 |
| rhallisey | sbezverk_, don't need to do it now | 14:45 |
| rhallisey | prechecks/logging will come soon | 14:45 |
| rhallisey | I'll make a bp | 14:45 |
| sbezverk_ | rhallisey: sounds good.. :-) | 14:46 |
| MarMat | sean-k-mooney yep, os_user ansible module has hependency on shade 1.5 and this was added to kolla-toolbox, so should run fine. | 14:46 |
| rhallisey | sbezverk_, thanks :) | 14:46 |
| HyperJohnGraham | sbezverk_: I have 90 minutes this morning before my first appointment. Do you have some time ? | 14:46 |
| sean-k-mooney | MarMat: ya i was likely still running an old kolla_toolbox. from a week or two ago. did not do a full rebuild just a rebuild of the ceph container when i hit the issue | 14:47 |
| *** lrensing has quit IRC | 14:47 | |
| openstackgerrit | Merged openstack/kolla-kubernetes: Adding kubectl commands to setup default namespace https://review.openstack.org/381349 | 14:51 |
| *** salv-orlando has joined #openstack-kolla | 14:51 | |
| sbezverk_ | HyperJohnGraham: Hi, yes | 14:55 |
| HyperJohnGraham | Ahh cool | 14:55 |
| *** unicell1 has quit IRC | 14:55 | |
| HyperJohnGraham | did you want to use webex again ? | 14:56 |
| sbezverk_ | HyperJohnGraham: yes please, here is the link: https://cisco.webex.com/join/sbezverk | 14:56 |
| *** salv-orlando has quit IRC | 14:56 | |
| *** MartinsG has quit IRC | 15:01 | |
| *** egonzalez90 has quit IRC | 15:07 | |
| *** dave-mccowan has quit IRC | 15:13 | |
| *** MarMat has quit IRC | 15:15 | |
| *** MarMat has joined #openstack-kolla | 15:16 | |
| *** diogogmt has joined #openstack-kolla | 15:21 | |
| sean-k-mooney | is anyone familar with kollas ha proxy configuration around? i was wonderin if it would be posible to extend kolla to allow ha proxy to be used only for internal traffic and disable it for the external traffic? | 15:21 |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla: Fix l3_agent.ini rendering error when DVR https://review.openstack.org/381919 | 15:22 |
| pbourke | bjolo: i have a fix for your neutron issue | 15:22 |
| pbourke | bjolo: can you give it a go? | 15:22 |
| pbourke | bjolo: https://review.openstack.org/381919 | 15:23 |
| *** dave-mccowan has joined #openstack-kolla | 15:30 | |
| *** dwalsh has quit IRC | 15:32 | |
| sean-k-mooney | by the way when is the stable/newton branch planned to be made? | 15:33 |
| *** eaguilar has quit IRC | 15:34 | |
| inc0 | kfox1111, sbezverk_ so nfs has issues, I think I'll try gluster instead | 15:35 |
| rhallisey | inc0, what issues are you seeing? | 15:36 |
| rhallisey | inc0, if we have nothing else, could go with ceph and have a script | 15:37 |
| inc0 | mariadb has issues with file locks | 15:37 |
| inc0 | rhallisey, but we don't have means to run this script | 15:37 |
| rhallisey | did it break your ceph? | 15:37 |
| inc0 | I haven't try;) | 15:38 |
| inc0 | what sbezverk_ was saying makes sense | 15:38 |
| inc0 | we need some way to run fencing script | 15:38 |
| *** salv-orlando has joined #openstack-kolla | 15:40 | |
| kfox1111 | rhallisey: or just use the kubectl evacuate to simulate node failure. it will do the right thing. | 15:41 |
| *** srwilkers has quit IRC | 15:41 | |
| kfox1111 | yes. I've not tried it with kolla-k8s, but have seen the issue first hand on another k8s system. | 15:41 |
| kfox1111 | if you kill the node, it wont release the rbd lock on volumes mounted. | 15:42 |
| kfox1111 | its kind of hard to say if its a ceph issue or a k8s one. | 15:42 |
| inc0 | I think it's k8s issue and ceph safety | 15:44 |
| *** mnasiadka has quit IRC | 15:45 | |
| kfox1111 | all rbd clients have the issue though. not just ceph. | 15:45 |
| kfox1111 | all rbd clients have the issue though. not just k8s. | 15:45 |
| kfox1111 | k8s just has enough extra knowledge to maybe work around it if ceph doesnt. | 15:46 |
| inc0 | gluster would give us both ha and might not suffer from same issue | 15:46 |
| * kfox1111 shrugs. | 15:46 | |
| kfox1111 | I think cephfs might work too. | 15:46 |
| kfox1111 | the issue is block based with locks on the dedicated volume. | 15:47 |
| *** matrohon has quit IRC | 15:47 | |
| inc0 | I'm pretty sure we can manually release lock on rbd, but what we would need is some logic that k8s could run on pod evac | 15:47 |
| kfox1111 | shared file systems don't have that issue. | 15:47 |
| kfox1111 | but they perform worse then dedicated block device. | 15:47 |
| kfox1111 | so kind of a tradeoff. | 15:47 |
| kfox1111 | yeah, manual lock evacuation works. I've trid it. | 15:48 |
| inc0 | issue is, how to release it? | 15:48 |
| kfox1111 | it really belongs in the code that probes kubelet aliveness and adds fencing too, as the node could still be alive. | 15:48 |
| kfox1111 | that parts firly simple. the rbd command has an unlock command. | 15:48 |
| *** dwalsh has joined #openstack-kolla | 15:48 | |
| kfox1111 | for the demo, I think we just write a "killnode.sh" script, that does: | 15:48 |
| inc0 | what I meant what will release it | 15:49 |
| kfox1111 | 1, ipmi power off, 2, drops the lock. | 15:49 |
| inc0 | (I'm sick hence my english is failing:() | 15:49 |
| *** huikang has joined #openstack-kolla | 15:49 | |
| kfox1111 | rbd lock remove imagename <lock-id> <locker> | 15:49 |
| kfox1111 | you get lock-id and locker from rbd lock list | 15:49 |
| inc0 | http://docs.ceph.com/docs/jewel/man/8/rbd/ --shared lock-tag | 15:50 |
| inc0 | how about this? | 15:50 |
| kfox1111 | k8s won't issue it without patching. | 15:51 |
| openstackgerrit | Waldemar Znoinski (wznoinsk) proposed openstack/kolla: WIP: setting basic structures for iscsi container https://review.openstack.org/381942 | 15:51 |
| inc0 | really? we can't specify flags of ceph rbd? | 15:52 |
| kfox1111 | nope. | 15:52 |
| kfox1111 | normally you don't need to. | 15:52 |
| *** daneyon has joined #openstack-kolla | 15:52 | |
| kfox1111 | and I don't think even rbd is doing it... | 15:52 |
| kfox1111 | I think the kernel issues the lock. | 15:52 |
| kfox1111 | I just think we should wait this one out. come up with something demo'able, but k8s will get a work around eventually. | 15:54 |
| kfox1111 | as is though, we only have 2 volumes, and I think we really could do with just one. | 15:54 |
| kfox1111 | and after we get mariadb clustering, even the lock issue won't be an issue, as there will be other members that will survive a node failure. | 15:55 |
| inc0 | what will happen if we issue lock remove before anything happends? | 15:56 |
| inc0 | right after rbd is created? | 15:56 |
| *** daneyon has quit IRC | 15:57 | |
| kfox1111 | the lock only gets created when its mounted., | 15:57 |
| kfox1111 | so when kubelet launches the pod, it rbd map voumename, to /dev/rbd0, which takes the lock, | 15:58 |
| kfox1111 | then it gets mounted into /var/lib/kubelet/volumes.... | 15:58 |
| kfox1111 | then the node dies, and nothing's there to pull the lock. | 15:58 |
| kfox1111 | so the next pod fails while trying to rbd map volumename | 15:58 |
| inc0 | but if rbd is mounted | 15:58 |
| *** dwalsh has quit IRC | 15:58 | |
| inc0 | we remove lock, rbd keeps being mounted | 15:59 |
| inc0 | after restart lock is already removed so we should be able to just remount | 15:59 |
| kfox1111 | yeah. so pulling a lock should only be done when you know for sure a node is dead. | 15:59 |
| kfox1111 | the issue is: | 15:59 |
| kfox1111 | node dies, it had the lock, it isn't there to release the lock. | 15:59 |
| inc0 | you sure? what if node is alive and well, aka we pull lock right after initial mount | 15:59 |
| kfox1111 | another node tries to launch the pod, tries to take the lock, its still there, it waits forever for the lock to release that won't ever get released. | 16:00 |
| inc0 | ok, let me just try what I'm thinking might be the issue;) | 16:00 |
| kfox1111 | all you have to do in that situation is delete the lock. k8s will recover it then. | 16:00 |
| *** lamt has joined #openstack-kolla | 16:01 | |
| *** mark-casey1 has joined #openstack-kolla | 16:03 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 16:03 |
| kfox1111 | so for the demo, the script to kill the node just has to release the lock too. everything shoudl work then. | 16:04 |
| *** huikang has quit IRC | 16:06 | |
| inc0 | k8s uses ssh to create mons? | 16:06 |
| *** huikang has joined #openstack-kolla | 16:06 | |
| kfox1111 | not sure I understand the question. | 16:07 |
| *** diogogmt_ has joined #openstack-kolla | 16:07 | |
| kfox1111 | s/k8s/kolla-kubernetes/; s/mons/volumes/ ? | 16:08 |
| inc0 | yes... | 16:08 |
| kfox1111 | ah. | 16:08 |
| inc0 | sorry, sick, language failing. | 16:08 |
| kfox1111 | no worries. :) | 16:08 |
| kfox1111 | that part is still in development. | 16:08 |
| *** diogogmt has quit IRC | 16:08 | |
| *** diogogmt_ is now known as diogogmt | 16:08 | |
| kfox1111 | the ps I'm working on literally right now: https://review.openstack.org/381041 | 16:08 |
| kfox1111 | does it a different way. | 16:08 |
| kfox1111 | I make an ceph-admin container, and kubectl exec stuff into it. | 16:09 |
| kfox1111 | much more k8s'ish. | 16:09 |
| inc0 | also when I specify ceph key...I base64 key that is base64? | 16:10 |
| kfox1111 | rhallisey: https://review.openstack.org/#/c/380848/ <- landed. you should have a hook now to start testing the ansible bits. | 16:10 |
| kfox1111 | k8s expects secrets to be base64'ed. it doesn't care what it is, it base64 decodes it on the other side. | 16:11 |
| *** dwalsh has joined #openstack-kolla | 16:11 | |
| kfox1111 | ceph expects a base64 encoded key too. | 16:11 |
| rhallisey | kfox1111, the main gate was suppose to run those bits | 16:11 |
| kfox1111 | so it needs to be double base64 encoded. | 16:11 |
| *** huikang has quit IRC | 16:11 | |
| kfox1111 | rhallisey: I don't follow. what main gate? | 16:12 |
| rhallisey | you added a second job template | 16:13 |
| kfox1111 | rhallisey: all it gives an extra flag now to the setup_get.sh. so we can branch there and fire off ansible instead of the shell script bits. | 16:13 |
| kfox1111 | yeah. it calls the same job in the end. it just had to be a different template because the args were a little different. | 16:14 |
| kfox1111 | and I didn't want to permutate all possible options. | 16:14 |
| kfox1111 | so its only 5 jobs, instead of 8. | 16:15 |
| rhallisey | that's fine | 16:16 |
| rhallisey | wfm | 16:17 |
| rhallisey | good way to test the anisble bits anyway | 16:17 |
| kfox1111 | yeah. and being merged in to the script that way, we can just branch around the parts we have in ansible, and continue to use shell for the rest, | 16:18 |
| kfox1111 | that way we can replace a piece at a time and still validate everything deploys ok. | 16:18 |
| sean-k-mooney | hi out of interest is there anything happening at the summit on the monday | 16:20 |
| sean-k-mooney | is there an operator day like austin? | 16:20 |
| *** david-lyle has joined #openstack-kolla | 16:21 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: Start testing the deployments https://review.openstack.org/380868 | 16:22 |
| kfox1111 | sbezverk_: I think the keepalived issue was maybe that br-ex's interface is down... not sure anything brought it up. | 16:23 |
| kfox1111 | goint to switch back to the external-ip resource to see if it fares any better. | 16:23 |
| kfox1111 | http://logs.openstack.org/68/380868/21/experimental/gate-kolla-kubernetes-deploy-centos-binary-ceph-nv/aaa7983/logs/ip.txt you can see br-ex is in state down and has no ip. but in the rest of the logs keepalived's running and seems happy. | 16:24 |
| *** openstackgerrit has quit IRC | 16:26 | |
| *** openstackgerrit has joined #openstack-kolla | 16:27 | |
| *** Pavo has quit IRC | 16:27 | |
| *** openstackgerrit has quit IRC | 16:28 | |
| *** openstackgerrit has joined #openstack-kolla | 16:28 | |
| *** openstackgerrit has quit IRC | 16:30 | |
| *** openstackgerrit has joined #openstack-kolla | 16:30 | |
| *** Pavo has joined #openstack-kolla | 16:33 | |
| sbezverk_ | kfox1111: makesure network_interface is defined | 16:33 |
| kfox1111 | I explicitly defined api_interface. | 16:34 |
| kfox1111 | bbiab | 16:35 |
| inc0 | sbezverk_, how can I debug pv creation? | 16:38 |
| sbezverk_ | inc0: any errors messages? | 16:41 |
| inc0 | I can't see any, but at the same time I don't know where to look:) | 16:41 |
| inc0 | so I created pv, it's there | 16:41 |
| inc0 | do rbd ls in ceph - > nothing | 16:41 |
| inc0 | mariadb bootstrap hangs on SuccessfulCreate Created pod | 16:42 |
| sbezverk_ | inc0: kubel get pvc --namesapce=kolla | 16:42 |
| sbezverk_ | inc0: kubel get pv --namesapce=kolla | 16:42 |
| sbezverk_ | on compute node where pod got started check /var/log/messages | 16:43 |
| sbezverk_ | mount | grep rbd | 16:43 |
| sbezverk_ | lsmod | grerp rbd | 16:43 |
| sbezverk_ | kubectl describe pod {mariadb pod name} --namespace=kolla | 16:44 |
| inc0 | hmm, let see | 16:45 |
| *** fguillot has quit IRC | 16:45 | |
| inc0 | hmm..yeah installation of ceph clients makes mounting rbd easier | 16:46 |
| sbezverk_ | inc0: :-) you bet | 16:47 |
| *** cj has joined #openstack-kolla | 16:51 | |
| cj | hey folks | 16:51 |
| cj | do we have a docker-compose setup available? | 16:51 |
| SamYaple | inc0: just fyi, kernel rbd mounting is not the best idea and doesnt work for voluems created in openstack due to the enabled features. best to stick with userspace | 16:51 |
| inc0 | SamYaple, not sure how k8s deals with it | 16:52 |
| inc0 | it might be userspace | 16:52 |
| inc0 | but still kernel module seems useful;) | 16:52 |
| *** eaguilar has joined #openstack-kolla | 16:53 | |
| inc0 | cj, docker compose?:) | 16:53 |
| inc0 | we were compose based like...2 years ago | 16:53 |
| kfox1111 | inc0: pv creation is just a record in k8s. | 16:54 |
| SamYaple | inc0: sure, but its old and outdated to the point that it actually _doesnt_ work for default rbd configuration in jewel. | 16:54 |
| cj | inc0: okay, that's no longer the new hotness? :-) | 16:54 |
| SamYaple | kernel only supports RBD_FEATURE_LAYERING | RBD_FEATURE_STRIPINGV2 | 16:54 |
| kfox1111 | you have to create/format the rbd volumes yourself. | 16:54 |
| kfox1111 | SamYaple: I think k8s onlyu supports kernel rbd at the moment. | 16:54 |
| SamYaple | kfox1111: ouch. sucks to be k8s | 16:55 |
| SamYaple | thats missing out on like all the good stuff | 16:55 |
| inc0 | cj, didn't really meet our requirements, openstack doesn't handle this things well, we moved on to ansible | 16:55 |
| *** msimonin1 has quit IRC | 16:55 | |
| cj | okay. I guess I'll look at ansible | 16:55 |
| SamYaple | https://github.com/torvalds/linux/blob/master/drivers/block/rbd.c#L119-122 | 16:56 |
| kfox1111 | SamYaple: yeah. It'll catch up eventually. | 16:56 |
| *** huikang has joined #openstack-kolla | 16:56 | |
| SamYaple | kfox1111: i know of no one working on the kernel module, but for what its worth, i did get it confirmed to me that the kernel module is _deprecated_ as the ML would have you believe | 16:56 |
| inc0 | kfox1111, you're saying that I need to create rbd myself? | 16:56 |
| kfox1111 | inc0: yes. | 16:57 |
| inc0 | meh | 16:57 |
| kfox1111 | inc0: there's the start of instructions here: https://review.openstack.org/#/c/381041/ | 16:57 |
| kfox1111 | meeting next. bbiab. | 16:57 |
| *** strigazi is now known as strigazi_AFK | 17:00 | |
| *** vhosakot has joined #openstack-kolla | 17:03 | |
| inc0 | kfox1111, ok, I wasn't aware of that, then our lock issue is easy | 17:05 |
| inc0 | just add --image-shared flag | 17:05 |
| inc0 | to rbd create | 17:05 |
| sbezverk_ | inc0: we have not tried that but I would assume other folks who hit this issue tried it | 17:06 |
| inc0 | don't assume too much;) | 17:06 |
| inc0 | when I get my ceph-backed stuff up I'll try | 17:06 |
| sbezverk_ | inc0: please let me know if it helps, will rebuild my volumes | 17:07 |
| *** eaguilar has quit IRC | 17:09 | |
| *** strigazi_AFK is now known as strigazi | 17:11 | |
| *** vhosakot has quit IRC | 17:15 | |
| *** vhosakot has joined #openstack-kolla | 17:15 | |
| *** strigazi is now known as strigazi_AFK | 17:22 | |
| inc0 | sbezverk_, kfox1111 look at this https://github.com/kubernetes/kubernetes/blob/3a42ec0b1df55c5ce96fe4e5c5c1b9fb0037f2c0/pkg/volume/rbd/rbd_util.go#L199 | 17:24 |
| rhallisey | interesting | 17:26 |
| *** vhosakot_ has joined #openstack-kolla | 17:27 | |
| *** tonanhngo has quit IRC | 17:29 | |
| *** vhosakot has quit IRC | 17:30 | |
| *** HyperJohnGraham has quit IRC | 17:37 | |
| *** eaguilar has joined #openstack-kolla | 17:47 | |
| *** fguillot has joined #openstack-kolla | 17:49 | |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla-kubernetes: Ansible support for Mariadb https://review.openstack.org/334255 | 17:51 |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla-kubernetes: Ansible support for Rabbitmq https://review.openstack.org/380796 | 17:51 |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla-kubernetes: Ansible workflow for Keystone https://review.openstack.org/380797 | 17:51 |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla-kubernetes: Ansible workflow for Glance https://review.openstack.org/380801 | 17:51 |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla-kubernetes: Workflow Gate Testing https://review.openstack.org/382021 | 17:51 |
| *** inc0 has quit IRC | 17:52 | |
| *** bjolo_ has joined #openstack-kolla | 17:53 | |
| *** Serlex has quit IRC | 17:53 | |
| *** tonanhngo has joined #openstack-kolla | 17:56 | |
| *** sdake has quit IRC | 17:58 | |
| *** tonanhngo has quit IRC | 18:00 | |
| *** dwalsh has quit IRC | 18:01 | |
| *** ayoung has joined #openstack-kolla | 18:01 | |
| *** harlowja has quit IRC | 18:03 | |
| *** vinkman has joined #openstack-kolla | 18:03 | |
| *** lrensing has joined #openstack-kolla | 18:05 | |
| *** LamT__ has quit IRC | 18:11 | |
| *** vinkman has left #openstack-kolla | 18:12 | |
| *** tonanhngo has joined #openstack-kolla | 18:14 | |
| kfox1111 | back. | 18:14 |
| *** tonanhngo has quit IRC | 18:14 | |
| kfox1111 | inc0, thats because k8s supports hving pods that share a read only volume. | 18:14 |
| kfox1111 | that's a correct statement. no locking is used/needed when everyone shares a read only volume. | 18:15 |
| kfox1111 | that doesn't help us though, as ours must be rw. | 18:15 |
| *** dwalsh has joined #openstack-kolla | 18:16 | |
| *** eaguilar has quit IRC | 18:18 | |
| *** HyperJohnGraham has joined #openstack-kolla | 18:22 | |
| *** haplo37 has quit IRC | 18:25 | |
| *** athomas has quit IRC | 18:25 | |
| kfox1111 | uh.... weird... | 18:26 |
| sbezverk_ | rhallisey: is there a way to test ansible workflows? | 18:26 |
| kfox1111 | http://logs.openstack.org/68/380868/22/experimental/gate-kolla-kubernetes-deploy-centos-binary-ceph-nv/3780c69/console.html | 18:26 |
| kfox1111 | seems like it got cut off half way through. | 18:27 |
| kfox1111 | sbezverk_: rhallisey: yeah, we could start adding in the workflow bits into setup_gate.sh | 18:27 |
| rhallisey | sbezverk_, yes | 18:27 |
| *** Pavo has quit IRC | 18:27 | |
| rhallisey | ansible-playbook -i workflows/ansible/inventory/all-in-one workflows/ansible/site.yml -e @/etc/kolla-kubernetes/kolla-kubernetes.yml | 18:27 |
| *** shardy is now known as shardy_afk | 18:27 | |
| sbezverk_ | rhallisey: oh nice, I was scratching my head on how to test it :-) | 18:28 |
| *** Pavo has joined #openstack-kolla | 18:28 | |
| *** unicell1 has joined #openstack-kolla | 18:29 | |
| *** g3ek has quit IRC | 18:29 | |
| kfox1111 | whaaa? | 18:29 |
| kfox1111 | http://logs.openstack.org/68/380868/22/experimental/gate-kolla-kubernetes-deploy-ubuntu-binary-ceph-nv/588148b/logs/openstack-catalog.txt | 18:29 |
| kfox1111 | no public endpoint!? | 18:29 |
| kfox1111 | arg.... | 18:30 |
| kfox1111 | we really need to break up our bootstrap jobs to 1 container, one thing. :/ | 18:30 |
| kfox1111 | the container seems to have failed, but told k8s it succeeded. | 18:30 |
| *** g3ek has joined #openstack-kolla | 18:30 | |
| kfox1111 | yeah. the container is doing too much, it doesn't check for failure, and the last thing succeeded... | 18:32 |
| sbezverk_ | kfox1111: we split already createdb/managedb/register endpoints | 18:33 |
| *** senk_ has quit IRC | 18:33 | |
| sbezverk_ | kfox1111: maybe the issue when you batch things together? | 18:33 |
| kfox1111 | no, I mean, split them to one job per task. create-endpoint-public, create-endpoint-internal, etc. | 18:33 |
| kfox1111 | can use a template to share all the code though. | 18:33 |
| kfox1111 | then if the one task fails, k8s will restart it. | 18:34 |
| *** haplo37 has joined #openstack-kolla | 18:34 | |
| kfox1111 | so maybe we make a common-create-endpoint.yml.j2 template, that takes in a var of what service, and what endpoint. | 18:35 |
| sbezverk_ | kfox1111: I noticed that restarting all tasks like createdb or maanagedb ot register endpoints usually causes bunch of troubles. as openstack does not handle well this operations of the same set of data | 18:35 |
| sean-k-mooney | has anyone else had issues with the rabbit mq contaienr constantly restarting on master? i and wznoinsk on our team have hit this | 18:36 |
| kfox1111 | sbezverk_: you mean rerunning them after they have run successfully, or just letting them rerun after they fail? | 18:36 |
| kfox1111 | splitting them up should help with the first case, as you only rerun the failed ones. | 18:36 |
| *** ccesario has quit IRC | 18:38 | |
| *** ccesario has joined #openstack-kolla | 18:39 | |
| *** tonanhngo has joined #openstack-kolla | 18:44 | |
| *** tonanhngo_ has joined #openstack-kolla | 18:46 | |
| *** srwilkers has joined #openstack-kolla | 18:47 | |
| *** tonanhngo_ has quit IRC | 18:48 | |
| *** dwalsh has quit IRC | 18:48 | |
| *** salv-orlando has quit IRC | 18:48 | |
| *** tonanhngo has quit IRC | 18:48 | |
| *** tonanhngo_ has joined #openstack-kolla | 18:48 | |
| *** vhosakot_ has quit IRC | 18:48 | |
| sbezverk_ | kfox1111: I meant running then if they by any reason were rescheduled | 18:48 |
| *** vhosakot has joined #openstack-kolla | 18:49 | |
| *** msimonin has joined #openstack-kolla | 18:50 | |
| sbezverk_ | kfox1111: do you want to take care of that? | 18:50 |
| *** ssl_ has joined #openstack-kolla | 18:51 | |
| *** ssl_ is now known as ssl123 | 18:51 | |
| kfox1111 | I can split them. should be too hard. | 18:51 |
| ssl123 | i'm running a test and made changes to nova.conf, how do i restart the nova compute service within the nova-compute container? | 18:52 |
| ssl123 | i can't do service nova-compute restart? | 18:52 |
| *** v1k0d3n has quit IRC | 18:54 | |
| *** v1k0d3n has joined #openstack-kolla | 18:54 | |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla-kubernetes: Workflow Gate Testing https://review.openstack.org/382021 | 18:55 |
| sean-k-mooney | pbourke: mdnadeem as an fyi i am hitting https://bugs.launchpad.net/kolla/+bug/1564773 on ubunut soucres with the head of master also | 18:56 |
| openstack | Launchpad bug 1564773 in kolla mitaka "Continously restarting rabbitmq container for CentOS" [High,Triaged] - Assigned to Md Nadeem (mail2nadeem92) | 18:56 |
| *** lrensing has quit IRC | 18:56 | |
| kfox1111 | oh.... this is weird... | 18:58 |
| kfox1111 | http://pastebin.com/XZTYdSrf | 18:58 |
| sean-k-mooney | kfox1111: you are seeing some of the endpoint multiple times ? | 18:59 |
| *** dwalsh has joined #openstack-kolla | 18:59 | |
| kfox1111 | no, a little weirder. let me show you. | 19:00 |
| kfox1111 | http://logs.openstack.org/68/380868/22/experimental/gate-kolla-kubernetes-deploy-ubuntu-binary-ceph-nv/588148b/logs/openstack-catalog.txt | 19:00 |
| mark-casey1 | ssl123: I'm looking in the logs for a discussion I had that touches on some of that. The brief version is: long-term changes need to be made prior to deploy, short term testing can be done by changing the configs on the server/VM that runs the nova container in a particular place (because these get mounted into the container) and using docker to restart the container | 19:00 |
| kfox1111 | I wonder if this is a race between parallel ansible jobs to build the endpoints. | 19:01 |
| kfox1111 | each doesn't see the service_endpoint, so they create one of their own. | 19:01 |
| *** ccesario has quit IRC | 19:01 | |
| kfox1111 | sometimes they run serially though, so things merge in that case. | 19:02 |
| sean-k-mooney | kfox1111: so is that why the glance image public enpoint is seperate form the admin and internal | 19:02 |
| sean-k-mooney | kfox1111: and why the cinderv2 service ans 3 seperate endpoints | 19:03 |
| *** vhosakot has quit IRC | 19:03 | |
| kfox1111 | yeah. I think so..... going to test this theory... | 19:03 |
| ssl123 | mark-casey1: i had some packages installed in the container.. if i restart the nova container, i assume i will lost those packages? | 19:03 |
| *** vhosakot has joined #openstack-kolla | 19:03 | |
| sean-k-mooney | kfox1111: where as keystone has a singel endpoint for interna public and admin | 19:03 |
| ssl123 | is there a way for me just to restart the nova-compute service within the container? | 19:03 |
| kfox1111 | say we have 2 endpoint creations about to go.... | 19:03 |
| kfox1111 | a and b. | 19:03 |
| kfox1111 | a starts up, sees no glance service registred, so decides to register its own. | 19:04 |
| sean-k-mooney | ssl123: restarting the container should not effeect the filesystem so your packages should still exist | 19:04 |
| mark-casey1 | ssl123: I don't think that's the way to do that but I'm not convinced those packages would go away | 19:04 |
| kfox1111 | b starts up sees no glance service registerd, so decides to register its own. | 19:04 |
| mark-casey1 | sean-k-mooney: right thanks :) | 19:04 |
| kfox1111 | b registers a new glance endpoint, getting back uuid 1111-1111-1111-1111 | 19:04 |
| kfox1111 | a registers a new glance endpoint, getting back uuid 2222-222-222-2222 | 19:05 |
| kfox1111 | a registers its url with that service uuid. | 19:05 |
| kfox1111 | b registers its url with its service uuid. | 19:05 |
| *** v1k0d3n has quit IRC | 19:05 | |
| kfox1111 | fun. | 19:05 |
| mark-casey1 | ssl123: you can certainly get direct help, like just now, but as a fallback this irc log has me dealing with some of this same topic a few months back: http://eavesdrop.openstack.org/irclogs/%23openstack-kolla/%23openstack-kolla.2016-04-20.log.html | 19:07 |
| sean-k-mooney | interesting. kfox1111 i guess the registrations will have to be done serially then to prevent this. | 19:07 |
| mark-casey1 | my question started "if I wanted to test a different value in nova.conf" and evolved into some discussion of how things work | 19:07 |
| kfox1111 | sean-k-mooney: yeah... I think we can still parallize them, so long as there are only one of each service running. so 3 parallel sets. one admin run, one internal, and one public. | 19:08 |
| *** v1k0d3n has joined #openstack-kolla | 19:08 | |
| sbezverk_ | kfox1111: they look independent, the only dependency if keystone up and running | 19:09 |
| kfox1111 | yeah. except for the service id creation dependency. | 19:09 |
| kfox1111 | the other way to do it would be a seperate job to create the service_id's. | 19:10 |
| sean-k-mooney | kfox1111: does the regeistration take place once per serverice per type(adming,internal,public) or is it also per contoler instance? | 19:10 |
| kfox1111 | then all the endpoint creation could be in parallel. | 19:10 |
| kfox1111 | sean-k-mooney: the former. | 19:10 |
| sean-k-mooney | ok well if its not liniarly dependent on the number of contolers then i dont think there will be a scaling reason to require a seperate step to precreate teh service_ids but it might be nicer to do that anyway | 19:11 |
| kfox1111 | yeah. | 19:12 |
| kfox1111 | that can be done in a followup ps though. maybe I just split and order in the first ps, and then we can do that in a followup. | 19:13 |
| sean-k-mooney | ok i have got to run. am before i do kfox1111 do you know when stable/newton will be cut in kolla? | 19:14 |
| kfox1111 | I'm afraid I may need to just do this all in one big ps. :/ | 19:14 |
| kfox1111 | sean-k-mooney: I don't know. :/ | 19:14 |
| sean-k-mooney | master seems to be kindof broken at the moment | 19:14 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: Start testing the deployments https://review.openstack.org/380868 | 19:16 |
| sean-k-mooney | in the last week or 2 i have hit 3 or 4 seperate bugs with ceph,rabbitmq,heat and magnum(that is fixed) | 19:17 |
| sean-k-mooney | anyway im an hour and a half late for going home so got to run | 19:18 |
| *** sean-k-mooney is now known as sean-k-mooneyAFK | 19:18 | |
| sbezverk_ | rhallisey: we should proabably remove multinode inventory since it is not applicable to kube at all. | 19:19 |
| kfox1111 | +1 | 19:20 |
| rhallisey | that's true | 19:20 |
| rhallisey | really only need one | 19:20 |
| rhallisey | since you're not deploy on machiens | 19:20 |
| kfox1111 | maybe just rename all-in-one 'kubernetes' or something? | 19:20 |
| kfox1111 | so your running with the "kubernetes" inventory? :) | 19:21 |
| sbezverk_ | or kube-api | 19:21 |
| kfox1111 | that works too. | 19:22 |
| sbezverk_ | since it is the entry point for us and api ip address would be there | 19:22 |
| kfox1111 | kollakube? :) | 19:22 |
| *** david-lyle has quit IRC | 19:22 | |
| kfox1111 | -i kollakube.... :) | 19:22 |
| sbezverk_ | kollakube is good | 19:22 |
| rhallisey | idc | 19:23 |
| rhallisey | any of those | 19:23 |
| kfox1111 | arg... like 3/4 of the time, the ceph tests fail in the ceph ps. | 19:23 |
| kfox1111 | every time pg 1.1 is stuck... | 19:24 |
| kfox1111 | its like ceph never fully deploys. | 19:24 |
| kfox1111 | there may be a network config issue there... | 19:24 |
| *** HyperJohnGraham has quit IRC | 19:26 | |
| sbezverk_ | kfox1111: hm but iscsi seems working ;-) maybe you should reconsider your view of it | 19:27 |
| *** vhosakot has quit IRC | 19:27 | |
| *** egonzalez90 has joined #openstack-kolla | 19:27 | |
| *** egonzalez90 has quit IRC | 19:27 | |
| kfox1111 | sbezverk_: nope. when you can forklift an lvm based system with zero downtime, I'll re'evaluate it. ;) | 19:28 |
| kfox1111 | the lvm backend is not ha. | 19:29 |
| kfox1111 | it can be made ha, but its harder to get ha then ceph. | 19:29 |
| *** daneyon has joined #openstack-kolla | 19:29 | |
| kfox1111 | I think this is more of a: the openstack test infrastructure is very heterogenious.... | 19:30 |
| kfox1111 | I think if I land on some nodes, it works, on others it doesnt. | 19:30 |
| kfox1111 | I've gotten several reports that the minikube docs work fine, which is ceph based. | 19:31 |
| kfox1111 | so its something different between minikube and the gate nodes. | 19:31 |
| *** daneyon has quit IRC | 19:34 | |
| sbezverk_ | kfox1111: for sure, I remember some time ago when kolla PS hit a specific gate host it was always failing.. | 19:34 |
| kfox1111 | ouch. | 19:34 |
| ssl123 | thanks mark-casey1 | 19:36 |
| *** ssl123 has quit IRC | 19:37 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 19:37 |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla-kubernetes: Workflow Gate Testing https://review.openstack.org/382021 | 19:39 |
| *** matrohon has joined #openstack-kolla | 19:40 | |
| *** shardy_afk is now known as shardy | 19:41 | |
| *** inc0 has joined #openstack-kolla | 19:41 | |
| inc0 | hello | 19:41 |
| inc0 | apperantly I'm alergic to Texas.. I just came back from doctor and I have alergic reaction for first time in my life | 19:42 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: Start testing the deployments https://review.openstack.org/380868 | 19:43 |
| rhallisey | inc0, ya I get that sometime when I go to different areas | 19:43 |
| kfox1111 | rhallisey: any chance we can do the functionizing in a different ps? We've got 3 ps's now all working on setup_gate. merge conflicts with the non indented version are going to be a bit painful by themselves. | 19:44 |
| kfox1111 | once all 3 are merged, then maybe we clean it all up? | 19:45 |
| kfox1111 | and maybe even break it up into multiple files... | 19:45 |
| rhallisey | kfox1111, not need to merge mine for a bit | 19:45 |
| rhallisey | kfox1111, i need the functions since there are 2 ways to run this | 19:45 |
| rhallisey | you guys an merge first | 19:45 |
| kfox1111 | k. | 19:45 |
| rhallisey | all the workflows aren't done anyway | 19:46 |
| rhallisey | this is easy wip | 19:46 |
| rhallisey | s/easy/early | 19:46 |
| kfox1111 | well, maybe just after k8s is deployed, just call out to a fuction for ansible and then exit, ignoring the rest? | 19:46 |
| kfox1111 | it could merge easier, and then we can functionize the rest. | 19:46 |
| rhallisey | I'm fine with rebasing a few times | 19:47 |
| rhallisey | it's no big deal | 19:47 |
| kfox1111 | k. | 19:47 |
| rhallisey | just add where you guys need to | 19:47 |
| kfox1111 | I haven't had rebases that big go well. but if you know how, great. :) | 19:47 |
| rhallisey | I've had my practice | 19:47 |
| rhallisey | :) | 19:47 |
| kfox1111 | cool. | 19:48 |
| kfox1111 | yeah. every time I do it I learn something new. :) | 19:48 |
| rhallisey | good stuff | 19:48 |
| *** bjolo_ has quit IRC | 19:49 | |
| *** bjolo_ has joined #openstack-kolla | 19:49 | |
| inc0 | kfox1111, https://review.openstack.org/#/c/381041/27 correct me if I'm wrong, but if I have ceph client somewhere I don't really need it right? | 19:50 |
| openstackgerrit | Mauricio Lima proposed openstack/kolla: [WIP] Add HNAS as Manila backend https://review.openstack.org/382054 | 19:50 |
| inc0 | this whole thing is just to have toolset to create stuff like pool or rbd image | 19:50 |
| *** salv-orlando has joined #openstack-kolla | 19:51 | |
| sbezverk_ | inc0: if you have ceph-common install you do not need it | 19:51 |
| kfox1111 | inc0: yeah. this ps is about smoothing out the deployment. | 19:52 |
| HyperJohnGraham_ | sbezverk_: i'm at my desk now | 19:52 |
| kfox1111 | so if you go and manuallyl create the rbd volumes elsewhere, or you preinstall ceph on all your k8s nodes, then you don't need it. | 19:52 |
| sbezverk_ | HyperJohnGraham_: 2 more minutes, need to grab a tea | 19:53 |
| kfox1111 | and its about adding gate checks for it all. | 19:53 |
| HyperJohnGraham_ | ok cool | 19:53 |
| kfox1111 | right now, the gate deploys mariadb/rabbitmq backed with hostpaths, which isn't ha. | 19:53 |
| kfox1111 | this is trying to get it past the finish line where everything's out of ceph and ha. | 19:53 |
| kfox1111 | meeting next. bbiab. | 19:54 |
| *** salv-orlando has quit IRC | 19:55 | |
| sbezverk_ | HyperJohnGraham_: https://cisco.webex.com/join/sbezverk | 19:57 |
| HyperJohnGraham_ | cool | 19:57 |
| *** salv-orlando has joined #openstack-kolla | 19:59 | |
| *** salv-orlando has quit IRC | 19:59 | |
| *** salv-orlando has joined #openstack-kolla | 20:00 | |
| *** bjolo_ has quit IRC | 20:01 | |
| *** shardy has quit IRC | 20:03 | |
| *** david-lyle has joined #openstack-kolla | 20:07 | |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla-kubernetes: Workflow Gate Testing https://review.openstack.org/382021 | 20:12 |
| *** lamt has quit IRC | 20:12 | |
| *** harlowja has joined #openstack-kolla | 20:15 | |
| *** HyperJohnGraham has joined #openstack-kolla | 20:16 | |
| *** david-lyle has quit IRC | 20:19 | |
| *** g3ek has quit IRC | 20:19 | |
| *** david-lyle has joined #openstack-kolla | 20:19 | |
| *** g3ek has joined #openstack-kolla | 20:20 | |
| *** Pavo has quit IRC | 20:27 | |
| *** lamt has joined #openstack-kolla | 20:29 | |
| *** Pavo has joined #openstack-kolla | 20:29 | |
| *** srwilkers has quit IRC | 20:30 | |
| *** srwilkers has joined #openstack-kolla | 20:30 | |
| *** coolsvap has quit IRC | 20:32 | |
| *** Pavo has quit IRC | 20:32 | |
| *** srwilkers has quit IRC | 20:33 | |
| *** Jeffrey4l_ has joined #openstack-kolla | 20:34 | |
| *** Jeffrey4l has quit IRC | 20:35 | |
| *** shardy has joined #openstack-kolla | 20:39 | |
| *** mark-casey1 has quit IRC | 20:41 | |
| *** v1k0d3n has quit IRC | 20:42 | |
| *** v1k0d3n has joined #openstack-kolla | 20:49 | |
| *** diogogmt_ has joined #openstack-kolla | 20:59 | |
| sbezverk_ | inc0: ping | 20:59 |
| inc0 | I'm here | 20:59 |
| sbezverk_ | inc0: from the top of your head, why do you think fetching ceph key task from ceph cluster might fail? | 21:00 |
| sbezverk_ | btw it is kolla, not kolla-kube | 21:00 |
| inc0 | ssh issue? | 21:01 |
| *** diogogmt has quit IRC | 21:01 | |
| *** diogogmt_ is now known as diogogmt | 21:01 | |
| inc0 | remember that it requires nodes to ssh to each other | 21:01 |
| inc0 | can | 21:01 |
| inc0 | can't remember which ones;) | 21:01 |
| sbezverk_ | inc0: everything else heka/toolsbox got deployed though | 21:02 |
| inc0 | so fetching keyrings is delegated task | 21:02 |
| inc0 | https://github.com/openstack/kolla/blob/master/ansible/roles/ceph/tasks/distribute_keyrings.yml#L7 | 21:03 |
| *** david-lyle has quit IRC | 21:04 | |
| sbezverk_ | inc0: I see | 21:04 |
| sbezverk_ | inc0: does it matter if it is the same machine? | 21:05 |
| inc0 | you have 1 mon? | 21:05 |
| sbezverk_ | inc0: yes | 21:06 |
| inc0 | hmm | 21:07 |
| inc0 | then it's probably not this, what is full error? | 21:07 |
| *** huikang has quit IRC | 21:08 | |
| *** huikang has joined #openstack-kolla | 21:09 | |
| sbezverk_ | inc0: error is coming from John | 21:09 |
| HyperJohnGraham_ | http://pastebin.com/p0Xx1dxJ | 21:09 |
| inc0 | https://github.com/openstack/kolla/blob/99897c5438f59b3fa40ade388e0eafe6a0fbfffb/doc/ceph-guide.rst#deploy-fails-with-fetching-ceph-keyrings--no-json-object-could-be-decoded | 21:11 |
| inc0 | :) | 21:11 |
| sbezverk_ | inc0: it seems ceph_mon is not running that is why it fails , but I cannot see why starting ceph_mon was skipped | 21:12 |
| inc0 | ^ | 21:12 |
| inc0 | docker volume ls | 21:13 |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla-kubernetes: Workflow Gate Testing https://review.openstack.org/382021 | 21:13 |
| HyperJohnGraham_ | http://pastebin.com/zgTCShQK | 21:13 |
| inc0 | HyperJohnGraham_, did you click the link I pasted?:) | 21:13 |
| *** huikang has quit IRC | 21:13 | |
| HyperJohnGraham_ | http://paste.openstack.org/show/584314/ | 21:14 |
| inc0 | HyperJohnGraham_, remove both ceph volumes before redeploying | 21:14 |
| inc0 | this error happends when you have stale volumes | 21:15 |
| sbezverk_ | inc0: thank you will try it | 21:15 |
| *** shardy has quit IRC | 21:15 | |
| *** matrohon has quit IRC | 21:15 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: Start testing the deployments https://review.openstack.org/380868 | 21:17 |
| kfox1111 | sbezverk_: have a look at this one when you get a chance... just the common jobs part. If I can get it all passing the gate checks, I might try and break it out to a seperate ps. it was just convenient to test here, as it has a lot more debuging in place. | 21:18 |
| *** ppalacios has quit IRC | 21:18 | |
| *** vhosakot has joined #openstack-kolla | 21:19 | |
| *** senk_ has joined #openstack-kolla | 21:20 | |
| HyperJohnGraham_ | http://paste.openstack.org/show/584317/ | 21:22 |
| *** schwicht has quit IRC | 21:24 | |
| *** eaguilar has joined #openstack-kolla | 21:25 | |
| sbezverk_ | inc0: it went further but not too much, could you check again? | 21:25 |
| sbezverk_ | kfox1111: the only concern is -e "{'service_auth':{{ global[serviceAuth] }}}" | 21:27 |
| sbezverk_ | does it get rendered ok? | 21:27 |
| kfox1111 | k. the gate check will verify that works or not. | 21:27 |
| kfox1111 | I believe it will. | 21:27 |
| inc0 | HyperJohnGraham_, when you see "container exited with non-zero code" next step should always be docker logs container-name | 21:28 |
| inc0 | so docker logs ceph_mon | 21:28 |
| sbezverk_ | inc0: we checked that and there was no any obivous cause of crash | 21:28 |
| inc0 | what is non-obvious one?:) logs plz | 21:29 |
| *** jheroux has quit IRC | 21:30 | |
| sbezverk_ | inc0: 2 seconds John stepped out | 21:30 |
| *** msimonin has quit IRC | 21:30 | |
| sbezverk_ | inc0: basically what we do it to use kolla-ansible to install ceph cluster and then to use it on John's kubernetes cluster which we just installed and configured | 21:31 |
| openstackgerrit | Christian Berendt proposed openstack/kolla: Create /var/log/kolla/rally before running rally-manage db create/upgrade https://review.openstack.org/382074 | 21:32 |
| inc0 | can I join in to webex? I try to do something similar | 21:32 |
| inc0 | I have ceph installed, I have k8s installed, but could use pointers on how to merge these two | 21:32 |
| sbezverk_ | absolutely!!!! | 21:32 |
| HyperJohnGraham_ | http://paste.openstack.org/show/584318/ | 21:33 |
| sbezverk_ | inc0: here is the link | 21:33 |
| sbezverk_ | https://cisco.webex.com/join/sbezverk | 21:33 |
| inc0 | guys, you need to remove these volumes | 21:33 |
| *** david-lyle has joined #openstack-kolla | 21:33 | |
| *** vhosakot_ has joined #openstack-kolla | 21:34 | |
| *** vhosakot has quit IRC | 21:35 | |
| *** huikang has joined #openstack-kolla | 21:36 | |
| openstackgerrit | Merged openstack/kolla: Fix l3_agent.ini rendering error when DVR https://review.openstack.org/381919 | 21:37 |
| *** senk_ has quit IRC | 21:38 | |
| *** MarMat has quit IRC | 21:39 | |
| *** MarMat has joined #openstack-kolla | 21:41 | |
| *** david-lyle has quit IRC | 21:42 | |
| *** vhosakot_ has quit IRC | 21:42 | |
| MarMat | anyone testing magnum currently? It seems broken to me due to that change in magnum https://github.com/openstack/magnum/commit/324f4aca7d194991a028068a9b8acef6f74e11c9 | 21:42 |
| openstackgerrit | Merged openstack/kolla: Fix sahara endpoint url https://review.openstack.org/381188 | 21:44 |
| *** diogogmt has quit IRC | 21:44 | |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla-kubernetes: Workflow Gate Testing https://review.openstack.org/382021 | 21:44 |
| *** david-lyle has joined #openstack-kolla | 21:47 | |
| *** david-lyle has quit IRC | 21:48 | |
| *** david-lyle has joined #openstack-kolla | 21:48 | |
| inc0 | HyperJohnGraham_, https://github.com/openstack/kolla/blob/master/doc/production-architecture-guide.rst#docker-configuration | 21:48 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: Start testing the deployments https://review.openstack.org/380868 | 21:49 |
| *** schwicht has joined #openstack-kolla | 21:50 | |
| *** dwalsh has quit IRC | 21:53 | |
| *** fguillot has quit IRC | 21:58 | |
| *** senk_ has joined #openstack-kolla | 22:01 | |
| *** eaguilar_ has joined #openstack-kolla | 22:02 | |
| *** eaguilar has quit IRC | 22:03 | |
| *** schwicht has quit IRC | 22:04 | |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla-kubernetes: Workflow Gate Testing https://review.openstack.org/382021 | 22:05 |
| *** lamt has quit IRC | 22:08 | |
| *** dave-mccowan has quit IRC | 22:11 | |
| *** daneyon has joined #openstack-kolla | 22:11 | |
| sbezverk_ | inc0: ceph auth get-key client.admin | 22:13 |
| sbezverk_ | AQBAMo1VqE1OMhAAVpERPcyQU5pzU6IOJ22x1w== | 22:13 |
| sbezverk_ | echo "AQBAMo1VqE1OMhAAVpERPcyQU5pzU6IOJ22x1w==" | base64 | 22:13 |
| sbezverk_ | QVFCQU1vMVZxRTFPTWhBQVZwRVJQY3lRVTVwelU2SU9KMjJ4MXc9PQo= | 22:13 |
| sbezverk_ | Edit your ceph-secret.yml with the base64 key: | 22:13 |
| sbezverk_ | apiVersion: v1 | 22:13 |
| sbezverk_ | kind: Secret | 22:13 |
| sbezverk_ | metadata: | 22:13 |
| sbezverk_ | name: ceph-secret | 22:13 |
| sbezverk_ | data: | 22:13 |
| sbezverk_ | key: QVFCQU1vMVZxRTFPTWhBQVZwRVJQY3lRVTVwelU2SU9KMjJ4MXc9PQo= | 22:13 |
| sbezverk_ | kubectl create -f secret/ceph-secret.yaml | 22:13 |
| *** daneyon has quit IRC | 22:16 | |
| *** huikang has quit IRC | 22:17 | |
| *** huikang has joined #openstack-kolla | 22:18 | |
| *** schwicht has joined #openstack-kolla | 22:20 | |
| *** huikang has quit IRC | 22:22 | |
| sbezverk_ | inc0: http://paste.openstack.org/show/584324/ | 22:24 |
| *** huikang has joined #openstack-kolla | 22:25 | |
| *** huikang has quit IRC | 22:30 | |
| *** schwicht has quit IRC | 22:39 | |
| kfox1111 | nice admin key. :) | 22:40 |
| *** senk_ has quit IRC | 22:40 | |
| inc0 | it's 256 of 2016 already? | 22:41 |
| inc0 | 0 256 016.M2 as warhammer 40k people would describe it | 22:44 |
| kfox1111 | :) | 22:45 |
| kfox1111 | you play? | 22:45 |
| *** eaguilar_ has quit IRC | 22:46 | |
| *** eaguilar has joined #openstack-kolla | 22:48 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: Start testing the deployments https://review.openstack.org/380868 | 22:49 |
| *** schwicht has joined #openstack-kolla | 22:49 | |
| kfox1111 | lets see if the broken out/common bootstrap jobs fixes the issue. :) | 22:49 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 22:53 |
| *** rhallisey has quit IRC | 22:55 | |
| *** senk_ has joined #openstack-kolla | 22:55 | |
| inc0 | kfox1111, pen and paper, video games and tons of books | 22:58 |
| *** sdake has joined #openstack-kolla | 22:58 | |
| inc0 | basically my favorite sf univere | 22:58 |
| *** salv-orlando has quit IRC | 23:01 | |
| kfox1111 | yeah. I really like 40k. just haven't had much time for it lately. :( | 23:03 |
| *** sdake_ has joined #openstack-kolla | 23:05 | |
| *** diogogmt has joined #openstack-kolla | 23:06 | |
| *** sdake has quit IRC | 23:09 | |
| *** diogogmt has quit IRC | 23:09 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: Start testing the deployments https://review.openstack.org/380868 | 23:09 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 23:13 |
| *** senk_ has quit IRC | 23:17 | |
| *** HyperJohnGraham has quit IRC | 23:18 | |
| *** rodrigo_pereira has quit IRC | 23:21 | |
| *** schwicht has quit IRC | 23:22 | |
| kfox1111 | nice... the idea does seem to have wings. :) | 23:26 |
| kfox1111 | it fixed the multiple service id's thing for sure. :) | 23:26 |
| kfox1111 | still running... will see if all the endpoints get created now. | 23:26 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 23:33 |
| sdake_ | hey peeeps | 23:38 |
| sdake_ | sorry internet is down | 23:38 |
| sdake_ | at parents atm | 23:38 |
| sdake_ | not sure what is busted | 23:38 |
| sdake_ | anyone blocked or need help? | 23:38 |
| kfox1111 | http://logs.openstack.org/68/380868/28/experimental/gate-kolla-kubernetes-deploy-centos-binary-ceph-nv/54cfea3/console.html almost there... so close... :) | 23:39 |
| kfox1111 | glance image uploaded, networks created, vm's starting to start... cli commands changed. :/ | 23:40 |
| *** senk_ has joined #openstack-kolla | 23:42 | |
| *** eaguilar has quit IRC | 23:46 | |
| *** awiddersheim has quit IRC | 23:47 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: Start testing the deployments https://review.openstack.org/380868 | 23:49 |
| *** awiddersheim has joined #openstack-kolla | 23:53 | |
| kfox1111 | theres something just really weird about ceph in the gate... | 23:53 |
| *** haplo37_ has quit IRC | 23:58 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!