| *** shashank_t_ has quit IRC | 00:01 | |
| sdake | you always know when an engine blows a man can't win em all | 00:02 |
|---|---|---|
| sdake | on that note - dinner time after torture session | 00:02 |
| *** deadnull_ has quit IRC | 00:03 | |
| *** yangyapeng has joined #openstack-kolla | 00:03 | |
| *** yangyape_ has quit IRC | 00:03 | |
| *** Pavo has joined #openstack-kolla | 00:05 | |
| *** goldyfruit has quit IRC | 00:07 | |
| *** schwicht has joined #openstack-kolla | 00:19 | |
| *** yangyape_ has joined #openstack-kolla | 00:20 | |
| *** yangyapeng has quit IRC | 00:20 | |
| *** dave-mccowan has joined #openstack-kolla | 00:22 | |
| *** Pavo has quit IRC | 00:24 | |
| *** yingjun has joined #openstack-kolla | 00:25 | |
| *** Pavo has joined #openstack-kolla | 00:26 | |
| *** fooliouno has joined #openstack-kolla | 00:32 | |
| openstackgerrit | Zeyu Zhu proposed openstack/kolla-kubernetes master: Remove useless modules https://review.openstack.org/450662 | 00:38 |
| *** duonghq_ is now known as duonghq | 00:38 | |
| duonghq | morning guys | 00:39 |
| duonghq | sdake, nice guide, I'm trying again with AIO first | 00:40 |
| duonghq | hope that it's ok | 00:40 |
| *** zhuzeyu has joined #openstack-kolla | 00:47 | |
| *** tovin07_ has joined #openstack-kolla | 00:59 | |
| *** Pavo has quit IRC | 01:02 | |
| *** MarginHu has joined #openstack-kolla | 01:04 | |
| *** yangyape_ has quit IRC | 01:06 | |
| *** cuongnv has joined #openstack-kolla | 01:09 | |
| *** eanylin has quit IRC | 01:11 | |
| *** adrian_otto has quit IRC | 01:18 | |
| *** zhubingbing_ has joined #openstack-kolla | 01:20 | |
| fooliouno | sdake: All my pods came up on a 2 node setup now. nova-compute is up and running as well. Thanks to you and everyone else who put the guide together. | 01:20 |
| *** david-lyle has joined #openstack-kolla | 01:24 | |
| fooliouno | Does anyone know where and how to specify a public endpoint for Horizon. It did not come up with one. | 01:26 |
| *** furlongm has quit IRC | 01:28 | |
| openstackgerrit | jimmygc proposed openstack/kolla-ansible master: Split globals.yml into mutiple part by service https://review.openstack.org/450106 | 01:32 |
| *** celo_r has joined #openstack-kolla | 01:33 | |
| openstackgerrit | jimmygc proposed openstack/kolla-ansible master: Split globals.yml into mutiple part by service https://review.openstack.org/450106 | 01:39 |
| *** caowei has joined #openstack-kolla | 01:41 | |
| *** shashank_t_ has joined #openstack-kolla | 01:44 | |
| *** eanylin has joined #openstack-kolla | 01:50 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Heat: add oslo_messaging_notifications config https://review.openstack.org/444515 | 01:56 |
| *** pramodrj07 has quit IRC | 01:58 | |
| *** MasterOfBugs has quit IRC | 01:58 | |
| *** dixiaoli has joined #openstack-kolla | 01:58 | |
| *** zhurong has joined #openstack-kolla | 02:04 | |
| *** yangyapeng has joined #openstack-kolla | 02:05 | |
| *** zhubingbing_ has quit IRC | 02:07 | |
| *** zhubingbing has joined #openstack-kolla | 02:10 | |
| *** unicell has quit IRC | 02:12 | |
| openstackgerrit | Zeyu Zhu proposed openstack/kolla-kubernetes master: Fix error in running-tests.rst https://review.openstack.org/451141 | 02:12 |
| sdake | fooliouno the guide works for you then? | 02:13 |
| sdake | fooliouno are you able to start a nova vm? | 02:14 |
| sdake | fooliouno we htink something is missing related to starting various helm charts and it hsoudl be pretty easy to find | 02:14 |
| *** fooliouno_ has joined #openstack-kolla | 02:16 | |
| fooliouno_ | sdake: openstack cli commands are being authenticated against the VIP address and they are failing. Troubleshooting them right now. | 02:16 |
| *** celo_r has quit IRC | 02:17 | |
| *** celo_r has joined #openstack-kolla | 02:17 | |
| *** fooliouno has quit IRC | 02:17 | |
| *** celo_r has quit IRC | 02:18 | |
| *** adrian_otto has joined #openstack-kolla | 02:22 | |
| *** goldyfruit has joined #openstack-kolla | 02:24 | |
| *** yangyapeng has quit IRC | 02:25 | |
| fooliouno_ | It appears that the keystone-admin address is being set to the kolla_internal_vip_address in globals.yml. Does that seem right? | 02:27 |
| *** adrian_otto has quit IRC | 02:27 | |
| *** shashank_t_ has quit IRC | 02:33 | |
| *** yangyapeng has joined #openstack-kolla | 02:33 | |
| *** david-lyle has quit IRC | 02:36 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla-ansible master: Updated from global requirements https://review.openstack.org/451017 | 02:40 |
| fooliouno_ | eanylin: Do you still set the kolla_internal_vip_address in globals.yml? | 02:47 |
| *** yuanying has quit IRC | 02:55 | |
| duonghq | fooliouno_, can you provide some log, it seems that same here | 02:55 |
| eanylin | fooliouno: I did set the internal_vip_address in globals.yml | 02:56 |
| eanylin | same IP as what I set for Kolla Ansible | 02:57 |
| fooliouno_ | eanylin: Thanks! I had it set, but had errors, so was just wondering. | 02:59 |
| fooliouno_ | duonghq: https://pastebin.com/FZiDP7G6 | 02:59 |
| eanylin | fooliouno: My initial set up has keystone internal same as keystone public | 02:59 |
| eanylin | I am trying your settings | 02:59 |
| eanylin | will see what happens | 02:59 |
| fooliouno_ | duonghq: The 10.124.11.200 is an unused IP that is in the same subnet as my 2 nodes. 10.124.11.29/23 | 03:01 |
| *** dave-mccowan has quit IRC | 03:03 | |
| fooliouno_ | I set the 10.124.11.200 as my kolla_internal_vip_address | 03:05 |
| duonghq | fooliouno_, did you try set VIP to one of kubeapi server? | 03:10 |
| duonghq | due to we are diabling haproxy | 03:10 |
| zhubingbing | hi guys | 03:11 |
| zhubingbing | ；） | 03:11 |
| duonghq | sup zhubingbing | 03:12 |
| openstackgerrit | zhongshengping proposed openstack/kolla-ansible master: Remove verbose option https://review.openstack.org/451155 | 03:13 |
| zhubingbing | nice meet you duonghq | 03:13 |
| fooliouno_ | duonghq: The IP of the node that the kube-apiserver is running on? | 03:13 |
| zhubingbing | your kolla-kubernetes cluster is be ok? | 03:13 |
| duonghq | zhubingbing, not yet | 03:14 |
| duonghq | last night I bring it to better state | 03:14 |
| zhubingbing | ;) | 03:14 |
| zhubingbing | i try to do it agin | 03:14 |
| duonghq | fooliouno_, yesss, did you try AIO deployment | 03:14 |
| zhubingbing | today | 03:14 |
| duonghq | zhubingbing, nice | 03:15 |
| fooliouno_ | I have a 2 node deployment with a kube master and the second node where all the kolla containers are deployed | 03:15 |
| duonghq | fooliouno_, ok | 03:15 |
| fooliouno_ | duonghq: I will try using that IP when I bring them up again. | 03:16 |
| *** yangyapeng has quit IRC | 03:17 | |
| duonghq | fooliouno_, nice | 03:18 |
| *** dixiaoli has quit IRC | 03:18 | |
| eanylin | fooliouno: I correct my earlier statement, that internal_vip was my kube-proxy container IP | 03:20 |
| eanylin | the one for kolla ansible has to be an IP that is not being used | 03:21 |
| eanylin | so they are not the same | 03:21 |
| *** dixiaoli has joined #openstack-kolla | 03:26 | |
| duonghq | zhubingbing, ping | 03:28 |
| zhubingbing | hear | 03:29 |
| openstackgerrit | zhongshengping proposed openstack/kolla-ansible master: Remove min_l3_agents_per_router option https://review.openstack.org/451159 | 03:29 |
| *** goldyfruit has quit IRC | 03:34 | |
| *** erlon has quit IRC | 03:35 | |
| *** david-lyle has joined #openstack-kolla | 03:37 | |
| fooliouno_ | eanylin: thanks for checking. Something in my configuration seems to be off. openstack cli commands dont work as they are directed to the unused IP. | 03:39 |
| *** zhubingbing has quit IRC | 03:42 | |
| eanylin | fooliouno: Ok. No problem. | 03:43 |
| eanylin | probably remove the config under /etc/kolla and re-generate? | 03:44 |
| fooliouno_ | eanylin: I tried that as well as cleaning up everything and bringing it up again. Still no luck. | 03:46 |
| eanylin | hmm | 03:47 |
| eanylin | what if you use keystone internal same as keystone public? | 03:47 |
| *** jrobinson has quit IRC | 03:52 | |
| fooliouno_ | I can try setting that to see if it works. Will give it a shot tomorrow. | 03:53 |
| eanylin | fooliouno: Ok | 03:53 |
| eanylin | Wasnt sure if you did the part on placement service/keystone endpoints creation, I think I figured out why it was crashing for me earlier | 03:55 |
| duonghq | anybody got "Failed to contact the endpoint at http://keystone-internal:5000/v3 for discovery." error? | 03:55 |
| eanylin | I can see the placement service and endpoints created in MariaDB now | 03:55 |
| *** jrobinson has joined #openstack-kolla | 03:56 | |
| eanylin | duonghq: Yeah, seeing similar error | 03:56 |
| eanylin | Failed to contact the endpoint at http://keystone-internal:5000/v3 for discovery. Fallback to using that endpoint as the base url. | 03:57 |
| eanylin | Unable to establish connection to http://keystone-internal:5000/v3/users?: HTTPConnectionPool(host='keystone-internal', port=5000): Max retries exceeded with url: /v3/users (Caused by NewConnectionError('< | 03:57 |
| duonghq | eanylin, have you resolved it? | 03:58 |
| eanylin | if keystone internal is set to keystone public URL, I think we will not see this error | 03:58 |
| eanylin | duonghq: Not yet. I tested this config since fooliouno seems to face similar problems | 03:58 |
| duonghq | eanylin, thanks | 03:58 |
| fooliouno_ | eanylin: I got the patch and also did a helm install for the placement service and cell0 | 03:59 |
| eanylin | I was trying to figure out the placement keystone creation failure for the most part | 03:59 |
| eanylin | since sbezverk says it worked for him | 03:59 |
| duonghq | eanylin, ok, I'm trying to deploy only keystone and connect to it, after it's successful I'll move on | 03:59 |
| eanylin | fooliouno: Yeah, but not sure if you did create the placement service and keystone endpoints or not | 04:00 |
| fooliouno_ | enaylin: which step does that. How to verify it | 04:00 |
| eanylin | duonghq: I think I used earlier guide with keystone internal = keystone public | 04:00 |
| duonghq | eanylin, I will try | 04:01 |
| eanylin | fooliouno: Some problems with my Openstack API calls due to the keystone-internal error | 04:01 |
| eanylin | I went direct to the mariadb | 04:01 |
| eanylin | to check | 04:01 |
| fooliouno_ | ok | 04:02 |
| eanylin | duonghq: Ok | 04:02 |
| duonghq | eanylin, the dns resolen keystone-internal correctly | 04:02 |
| eanylin | duonghq: Ok, I set the kube-dns IP in my resolv.conf but wasnt able to resolve | 04:02 |
| eanylin | so it worked for you? | 04:03 |
| duonghq | eanylin, which pod are you using for test this? | 04:03 |
| eanylin | kolla | 04:03 |
| eanylin | sorry, thats the namespace | 04:03 |
| duonghq | I'm using busybox, it's resolve correctly | 04:03 |
| eanylin | ah | 04:04 |
| eanylin | ok | 04:04 |
| eanylin | maybe I can try that too | 04:04 |
| sdake | eanylin sbezverk hasn't run throug hthe guide - so its unclear that he has reproducibel steps - he knwos how this stuff works in detial and he uses the gate scripts as a guide | 04:04 |
| fooliouno_ | The OS_AUTH_URL in my admin-openrc was pointing to the kolla_internal_vip_address. I changed that to my node's public IP. Now the openstack commands try to access keystone-internal and the name resolution fails. | 04:05 |
| sdake | i'm glad we are making really good progress though :) | 04:05 |
| duonghq | fooliouno_, so you stuck at some point as me | 04:06 |
| fooliouno_ | sdake: Agree .. the new docs certainly helped speed up the process | 04:06 |
| fooliouno_ | duonghq: looks like it :) | 04:06 |
| kolla-slack | <vhosakot> @sdake did not get time to test your guide today… will test tomorrow/this week for sure. | 04:06 |
| sdake | fooliouno_ versus guessing or reading the gate :) | 04:06 |
| sdake | vhsosakot I think we are going to merge something soon | 04:07 |
| sdake | and then iterate on that | 04:07 |
| kolla-slack | <vhosakot> @sdake cool +1 | 04:07 |
| duonghq | sdake, so seem that I and fooliouno_ got same issue, | 04:07 |
| eanylin | sdake: I think I figured out how to get placement service/keystone endpoints created | 04:07 |
| sdake | so its rendered to work from | 04:07 |
| fooliouno_ | sdake: yep. definitely saved a lot of time for me | 04:07 |
| eanylin | it has got to do with sequencing | 04:07 |
| eanylin | on which microservice to start first | 04:07 |
| kolla-slack | <vhosakot> @sdake yep, good to get something merged soon, and then improve as needed. | 04:07 |
| sdake | yup - this i s why it needs to be inth nova service chart | 04:08 |
| eanylin | sdake: Yeah, I went and read the openstack installation document | 04:08 |
| eanylin | for placement | 04:08 |
| kolla-slack | <vhosakot> @fooliouno did you test @sdake’s guide? if so, how far did you go? were you able to get all OpenStack pods up? | 04:08 |
| sdake | until that time (which is easy to get to) we can just add some waits on the various services | 04:08 |
| eanylin | sdake: You want me to put those steps in or? | 04:08 |
| sdake | eanylin yes plz | 04:09 |
| fooliouno_ | vhosakot: Yes .. pretty much at the end. All pods up. issue with running openstack commands. | 04:09 |
| eanylin | sdake: Ok. I will leave comments | 04:09 |
| kolla-slack | <vhosakot> @fooliouno_ nice! :slightly_smiling_face: | 04:09 |
| eanylin | duonghq: busy box resolves for me too | 04:09 |
| *** zhubingbing has joined #openstack-kolla | 04:09 | |
| duonghq | eanylin, nice | 04:09 |
| sdake | eanylin surprised you ahve een able to replicate the deploy from the docs :) | 04:09 |
| duonghq | so we don't what's wrong | 04:09 |
| sdake | eanylin or maybe you helped make the docs more then replicate it :) | 04:10 |
| duonghq | *don't know | 04:10 |
| duonghq | kollo container doesn't have nslookup | 04:10 |
| sdake | duonghq ok - so eveyrone has the same problem then which is more specifically what? | 04:10 |
| duonghq | sdake, I'm finding what is cannot contact to keystone-internal | 04:11 |
| sdake | duonghq the placement api is needed or nothing wil lwork | 04:11 |
| sdake | ok well thats a different probelm then the placement api | 04:11 |
| duonghq | sdake, even keystone cannot work by itself? | 04:11 |
| sdake | duonghq let me asks yo uthis | 04:12 |
| sdake | can you run openstack catalog list pls? | 04:12 |
| duonghq | sdake, I see something interesting here | 04:12 |
| duonghq | [sorry for raw paste] | 04:12 |
| duonghq | | keystone | identity | RegionOne | | 04:12 |
| duonghq | | | | public: http://192.168.122.115:5000/v3 | | 04:12 |
| duonghq | | | | RegionOne | | 04:12 |
| duonghq | | | | internal: http://keystone-internal:5000/v3 | | 04:12 |
| duonghq | | | | RegionOne | | 04:12 |
| duonghq | | | | admin: http://192.168.122.115:35357/v3 | | 04:12 |
| sdake | winning | 04:12 |
| sdake | ok thats really god | 04:13 |
| sdake | good | 04:13 |
| sdake | i ran int ot his problem about 3 weeks ago | 04:13 |
| sdake | i wish i had etter notes about what i di ot solve it | 04:13 |
| sdake | i knwo it involved setting a value in cloud.yaml | 04:13 |
| duonghq | sdake, what is the expected output? | 04:13 |
| sdake | the fact that keystone spit out values means keystone works | 04:14 |
| sdake | catalog list comes from keystone unauthenticated | 04:14 |
| sdake | e.g. port 5000 of keystone | 04:14 |
| sdake | 35357 is the authenticated port | 04:14 |
| sdake | what error do yo usee when you run keystone endpoint list? | 04:15 |
| duonghq | kolla@master:~/kolla-bringup$ openstack endpoint list | 04:16 |
| duonghq | Failed to contact the endpoint at http://keystone-internal:5000/v3 for discovery. Fallback to using that endpoint as the base url. | 04:16 |
| sdake | duonghq how did you create your keystone rc file? | 04:16 |
| eanylin | sdake: Updated, left the comment for nova placement | 04:16 |
| sdake | eanylin cool will integreate shortly | 04:16 |
| duonghq | sdake, using kolla-ansible post-deploy | 04:16 |
| eanylin | sdake: Haha :) | 04:16 |
| eanylin | sdake: Cool | 04:17 |
| *** schwicht has quit IRC | 04:17 | |
| eanylin | sdake: I think we are left with the cell0 puzzle now | 04:17 |
| sdake | duonghq in the tools directory is something that creates an openrc | 04:17 |
| sdake | can you try it | 04:17 |
| sdake | with ext as the termination | 04:18 |
| sdake | moment | 04:18 |
| duonghq | sdake, you mean ./build_local_admin_keystonerc.sh | 04:18 |
| kolla-slack | <vhosakot> duonghq can you try "openstack user list --domain default” ? | 04:19 |
| *** bmace has quit IRC | 04:19 | |
| sdake | duonghq yes | 04:19 |
| sdake | please run that with ext as a paraemteter | 04:19 |
| sdake | then source the file it creates | 04:19 |
| sdake | which i htink is ~/keystone_adminrc | 04:20 |
| sdake | or somethign like that | 04:20 |
| sdake | i may ahve used that method | 04:20 |
| sdake | rather hten post-deploy | 04:20 |
| duonghq | sdake, you mean with arg xext? | 04:20 |
| sdake | just ext | 04:21 |
| sdake | xext==x the x tests for an empty value | 04:21 |
| sdake | shell ftl | 04:21 |
| duonghq | ah, | 04:21 |
| duonghq | sdake, voila | 04:22 |
| sdake | did that work? | 04:22 |
| duonghq | yes | 04:22 |
| sdake | nice! | 04:22 |
| sdake | who else was suffering that problem? | 04:22 |
| sdake | plase leave a comment in the docs review | 04:22 |
| sdake | I will address that now | 04:22 |
| fooliouno_ | eanylin: saw your comment on order of placement microservices. Should all of them be run after nova-control and nova-compute? | 04:22 |
| eanylin | fooliouno, I havent really test the order but I think so, its probably ok to run after that | 04:23 |
| eanylin | I saw in some forum that they manually do it to upgrade their environment | 04:24 |
| eanylin | duonghq: I think I found a way to get round that. Not very elegant though | 04:24 |
| fooliouno_ | ok .. we were not running all of them earlier. | 04:24 |
| sdake | eanylin souds like he got around it | 04:24 |
| eanylin | fooliouno: I wanted to run those but I keep getting keystone endpoint create fails | 04:25 |
| sdake | and it is elegant | 04:25 |
| eanylin | so was investigating why it was not working | 04:25 |
| eanylin | cool, so it worked for duonghq | 04:25 |
| sdake | yup running that shell script and sourcing keystone_adminrc are what i did - and I docuemnted the wrong thing | 04:26 |
| eanylin | oh | 04:26 |
| eanylin | I went and use nslookup to find that IP | 04:26 |
| duonghq | eanylin, sdake the "new" rc file use admin endpoint port | 04:26 |
| duonghq | with a couple of extra settings | 04:27 |
| sdake | yup crafting an openrc file is a little difficult | 04:27 |
| duonghq | but PS1=... is redundant | 04:27 |
| sdake | post-deploy used to work fine | 04:27 |
| sdake | not sure why it doesnt' work now | 04:27 |
| sdake | doesn't matter as we are getting rid of all that | 04:27 |
| eanylin | so post deploy is not the right one to use then? | 04:27 |
| sdake | (in kolla-kubernetes) | 04:28 |
| duonghq | eanylin, for kolla-k8s, seem that | 04:28 |
| sdake | a python implementation will probably hit for that ile | 04:28 |
| duonghq | the bigest different is the port | 04:28 |
| eanylin | duonghq: I see | 04:28 |
| sdake | duonghq mind pasting both versions | 04:28 |
| duonghq | sure | 04:28 |
| eanylin | duonghq sdake: The new method works | 04:29 |
| duonghq | sdake, http://paste.openstack.org/show/604608/ | 04:29 |
| eanylin | is PS1 redundant? | 04:30 |
| duonghq | eanylin, yes | 04:30 |
| *** dixiaoli has quit IRC | 04:30 | |
| eanylin | duonghq: Seems so | 04:30 |
| eanylin | I unset that | 04:31 |
| eanylin | it works too | 04:31 |
| duonghq | eanylin, yup, it does not have any business here | 04:31 |
| eanylin | agree | 04:31 |
| eanylin | so guess just need to update the guide to run this command instead of the old one | 04:31 |
| sdake | wonder what this is: build_local_admin_keystonerc.sh | 04:31 |
| kolla-slack | <vhosakot> duonghq did the keystone CLI work? | 04:32 |
| sdake | export OS_INTERFACE=internal | 04:32 |
| sdake | rather | 04:32 |
| duonghq | vhosakot: I cannot install keystone CLI | 04:32 |
| duonghq | but openstack user list work | 04:33 |
| sdake | duonghq use openstack client | 04:33 |
| fooliouno_ | duonghq: Whats the IP in your OS_AUTH_URL pointing to | 04:33 |
| sdake | keystone client is busted | 04:33 |
| duonghq | sdake, sure | 04:33 |
| duonghq | fooliouno_, I'm using AIO, so it's the node ip | 04:33 |
| kolla-slack | <vhosakot> I meant, sorry openstack CLI, I know keystone is deprecated :slightly_smiling_face: | 04:33 |
| kolla-slack | <vhosakot> I meant keystone CLI is deprecated :slightly_smiling_face: | 04:34 |
| duonghq | vhosakot: we know what you meant :P | 04:34 |
| eanylin | openstack service list works too | 04:34 |
| eanylin | should be good | 04:34 |
| duonghq | glance works fine | 04:35 |
| kolla-slack | <vhosakot> cool, keystone auth works! :slightly_smiling_face: | 04:35 |
| duonghq | vhosakot: nice | 04:36 |
| kolla-slack | <vhosakot> no, I have not tried it yet :slightly_smiling_face: I was happy for you guys :slightly_smiling_face: | 04:36 |
| duonghq | :) | 04:37 |
| eanylin | :) | 04:37 |
| duonghq | eanylin, you are Anthony Lin? | 04:37 |
| eanylin | duonghq: Yes, I am | 04:37 |
| duonghq | eanylin, placement api microservice should be installed after all other service? | 04:38 |
| eanylin | hmm, actually I havent tested, so I am not sure too | 04:38 |
| eanylin | I know how to get it into the mariaDB now | 04:38 |
| eanylin | it wasnt getting created earlier | 04:38 |
| duonghq | eanylin, ya, | 04:38 |
| eanylin | if I dont follow certain sequencing | 04:38 |
| fooliouno_ | sdake: enaylin: duonghq: The new keystonerc works for me as well! :) | 04:39 |
| duonghq | fooliouno_, great | 04:39 |
| *** yuanying has joined #openstack-kolla | 04:39 | |
| eanylin | fooliouno: Cool :) | 04:39 |
| *** unicell has joined #openstack-kolla | 04:39 | |
| eanylin | duonghq: I think we can try the sequence of creation of the nova services | 04:40 |
| duonghq | eanylin, can you explain? | 04:40 |
| eanylin | to see if it makes any difference | 04:40 |
| duonghq | I'm installing neutron | 04:40 |
| duonghq | head to nova now | 04:40 |
| eanylin | like create nova-control,compute, then placement and then cell0 | 04:41 |
| eanylin | I think that might be the right sequence | 04:41 |
| eanylin | since cell0 depends on placement | 04:41 |
| eanylin | I think | 04:41 |
| fooliouno_ | Any idea how to create a public endpoint for Horizon so that it is accessible directly via browser from the outside | 04:41 |
| *** gkadam has joined #openstack-kolla | 04:42 | |
| eanylin | fooliouno: We were trying port forward earlier | 04:42 |
| eanylin | like port forward port 80 of horizon pod | 04:42 |
| fooliouno_ | ok | 04:42 |
| eanylin | but I think there might be some problems with it, saw some issues earlier | 04:42 |
| sdake | there is an easier way to make the port external | 04:43 |
| duonghq | fooliouno_, just one setting in cloud.yaml | 04:43 |
| *** bmace has joined #openstack-kolla | 04:44 | |
| fooliouno_ | duonghq: whats the setting | 04:44 |
| duonghq | fooliouno_, you can see helm/all_values.yaml | 04:44 |
| duonghq | it's default one | 04:44 |
| fooliouno_ | Let me check | 04:44 |
| duonghq | and cloud.yaml will override that | 04:44 |
| eanylin | oh | 04:45 |
| eanylin | so just need to make it external and it will override? | 04:45 |
| sdake | eanylin looking for optoin moment | 04:46 |
| duonghq | one more thing: kubernetes does not allow we use node_port with port is 80 | 04:46 |
| duonghq | by default in kubeadm | 04:46 |
| fooliouno_ | port_external should be set to true? | 04:46 |
| eanylin | I did this | 04:47 |
| duonghq | fooliouno_, yup | 04:47 |
| eanylin | [root@kube03 ~]# kubectl port-forward horizon-3565390796-4w349 -n kolla 80 | 04:47 |
| eanylin | Forwarding from 127.0.0.1:80 -> 80 | 04:47 |
| eanylin | Forwarding from [::1]:80 -> 80 | 04:47 |
| eanylin | Handling connection for 80 | 04:47 |
| eanylin | Handling connection for 80 | 04:47 |
| eanylin | Handling connection for 80 | 04:47 |
| eanylin | so you can see it will also work | 04:47 |
| *** shashank_t_ has joined #openstack-kolla | 04:47 | |
| sdake | eanylin thats too complicated :) - there is an eaiser way | 04:47 |
| sdake | glance-api-svc: | 04:47 |
| sdake | global: | 04:47 |
| sdake | kolla: | 04:47 |
| sdake | glance: | 04:47 |
| sdake | api: | 04:47 |
| sdake | all: | 04:47 |
| sdake | port: 9292 | 04:47 |
| sdake | node_port: 9292 | 04:47 |
| sdake | port_external: false | 04:47 |
| sdake | node_port_enabled: false | 04:47 |
| sdake | in cloud.yaml set port-external: true | 04:48 |
| eanylin | sdake: Ok, let me check | 04:48 |
| sdake | port-external | 04:48 |
| sdake | port_external | 04:48 |
| sdake | gah - keyboard | 04:48 |
| sdake | dont do it in all_values.yaml - instead cloud.yaml overrides thsoe defaults | 04:48 |
| fooliouno_ | yup .. I was looking there for the field. | 04:48 |
| sdake | all_values.yaml contains all fields | 04:49 |
| sdake | cloud.yaml is a subset of overrides from the defaults | 04:49 |
| duonghq | need leaving now, BRB | 04:49 |
| sdake | later duonghq | 04:49 |
| spsurya | sdake: can you please add me too as Co-Author in this preparation https://review.openstack.org/#/c/447356/ ? | 04:49 |
| fooliouno_ | later duonghq. Thansk for the help | 04:49 |
| sdake | spsurya sure | 04:49 |
| eanylin | later duonghq | 04:50 |
| spsurya | sdake: thanks | 04:50 |
| eanylin | horizon: | 04:50 |
| eanylin | all: | 04:50 |
| eanylin | node_port: 80 | 04:50 |
| eanylin | node_port_enabled: false | 04:50 |
| eanylin | port: 80 | 04:50 |
| eanylin | port_external: true | 04:50 |
| eanylin | should work? | 04:50 |
| fooliouno_ | I think so. I made the same change, but will try it tomorrow. | 04:52 |
| sdake | spsurya what is your email address - gerrit isn't popping it up for me for some reason | 04:52 |
| spsurya | sdake: surya.singh@nectechnologies.in | 04:53 |
| fooliouno_ | I'm gonna call it a night. Thanks everyone! | 04:53 |
| spsurya | please use this | 04:54 |
| spsurya | sdake: | 04:54 |
| eanylin | fooliouno: That is the right config | 04:54 |
| eanylin | I just tested | 04:54 |
| fooliouno_ | you are fast :) | 04:55 |
| eanylin | take note of the indentation though, for the yaml file | 04:55 |
| eanylin | :) | 04:55 |
| eanylin | I delete my horizon and create again :D | 04:55 |
| eanylin | sdake: I am going to update the cloud.yaml to include this for Horizon | 04:57 |
| sdake | eanylin i hink that would be better then what i mentioned | 04:58 |
| sdake | eanylin yo ushouldn't have to override node_port or port por node_port-enabled | 04:58 |
| sdake | shoudl be able to do horizon: | 04:59 |
| sdake | all: | 04:59 |
| sdake | port_external: true | 04:59 |
| sdake | we are after a minimal working cloud.yaml | 04:59 |
| eanylin | sdake: Ok | 05:00 |
| eanylin | makes sense | 05:00 |
| eanylin | rather than repeating | 05:01 |
| eanylin | since the rest are default | 05:01 |
| *** iceyao has joined #openstack-kolla | 05:02 | |
| eanylin | updated | 05:03 |
| spsurya | sdake: hope you got surya.singh@nectechnologies.in | 05:03 |
| sdake | spsurya yup - surya is your first name? | 05:04 |
| *** obre_ has joined #openstack-kolla | 05:06 | |
| *** obre has quit IRC | 05:07 | |
| *** kwazar has quit IRC | 05:07 | |
| *** markus-k has quit IRC | 05:07 | |
| *** kwazar_ has joined #openstack-kolla | 05:07 | |
| *** markus-k_ has joined #openstack-kolla | 05:07 | |
| spsurya | sdake: Surya Prakash is the first name | 05:10 |
| sdake | build_example_yaml.py doesn't work | 05:11 |
| spsurya | and Singh is the last | 05:11 |
| sdake | spsurya thanks | 05:11 |
| sdake | just want to get hte name correct in the review | 05:11 |
| spsurya | sdake: nic <spsurya> is also fine :) | 05:12 |
| sdake | spsurya has to be formatted correctly for gerrit i think | 05:13 |
| *** iceyao has quit IRC | 05:13 | |
| sdake | welli just screewed up my work on the review | 05:14 |
| sdake | i think i am headed to bed | 05:14 |
| sdake | perhaps people can add their comments | 05:14 |
| sdake | and I'll address in the morning | 05:14 |
| *** iceyao has joined #openstack-kolla | 05:14 | |
| spsurya | sdake: sure | 05:14 |
| spsurya | BTW night | 05:14 |
| *** iceyao has quit IRC | 05:15 | |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Deployment Guide Documentation https://review.openstack.org/447356 | 05:22 |
| *** adrian_otto has joined #openstack-kolla | 05:26 | |
| *** dixiaoli has joined #openstack-kolla | 05:31 | |
| *** dixiaoli has quit IRC | 05:34 | |
| *** sayantan_ has joined #openstack-kolla | 05:46 | |
| *** shashank_t_ has quit IRC | 05:54 | |
| *** zhurong has quit IRC | 05:54 | |
| *** shashank_t_ has joined #openstack-kolla | 05:55 | |
| *** dixiaoli has joined #openstack-kolla | 05:58 | |
| *** shashank_t_ has quit IRC | 05:59 | |
| *** jrobinson has quit IRC | 06:01 | |
| openstackgerrit | Surya Prakash (spsurya) proposed openstack/kolla-kubernetes master: Deployment Guide Documentation https://review.openstack.org/447356 | 06:02 |
| openstackgerrit | Zeyu Zhu proposed openstack/kolla-kubernetes master: Add a script to generate configuration https://review.openstack.org/451199 | 06:12 |
| *** jaosorior has joined #openstack-kolla | 06:14 | |
| eanylin | sdake: Left a comment on create-cell-job, which I think is related to cell0. Take a look when you are up in the morning. Thanks. | 06:14 |
| *** Serlex has joined #openstack-kolla | 06:21 | |
| *** pcaruana has joined #openstack-kolla | 06:24 | |
| *** tovin07 has joined #openstack-kolla | 06:25 | |
| *** caowei has quit IRC | 06:26 | |
| *** caowei has joined #openstack-kolla | 06:27 | |
| *** targon has joined #openstack-kolla | 06:28 | |
| *** bjolo has joined #openstack-kolla | 06:41 | |
| Jeffrey4l | sbezverk, around? | 06:46 |
| *** oanson has quit IRC | 06:48 | |
| openstackgerrit | chenyingnan proposed openstack/kolla-ansible master: Allow nfs, rpcbind and mountd service on server firewall https://review.openstack.org/451210 | 06:49 |
| duonghq | anybody got Target WSGI script not found or unable to stat: /usr/share/openstack-dashboard in horizon (kolla-k8s) | 06:49 |
| *** matrohon has joined #openstack-kolla | 06:55 | |
| *** athomas has joined #openstack-kolla | 06:57 | |
| *** matrohon has quit IRC | 07:01 | |
| openstackgerrit | Surya Prakash (spsurya) proposed openstack/kolla-kubernetes master: Deployment Guide Documentation https://review.openstack.org/447356 | 07:06 |
| *** sayantan_ has quit IRC | 07:18 | |
| *** sayantan_ has joined #openstack-kolla | 07:19 | |
| *** sayantan_ has quit IRC | 07:23 | |
| duonghq | spsurya, ping | 07:23 |
| spsurya | duonghq: hi.. | 07:24 |
| duonghq | about kobernetes-deploymen guide | 07:24 |
| duonghq | why do you remove line 102 | 07:24 |
| duonghq | I mean the slash is redundant, not whole comment | 07:25 |
| duonghq | *command | 07:25 |
| *** jmccarthy has joined #openstack-kolla | 07:25 | |
| spsurya | duonghq: ohh | 07:25 |
| spsurya | duonghq: please update that :( | 07:25 |
| spsurya | duonghq: by mistake *dd* done that :) i guess | 07:27 |
| spsurya | i was removing the below line once | 07:28 |
| duonghq | spsurya, I will update | 07:28 |
| spsurya | duonghq: | 07:29 |
| spsurya | thanks | 07:29 |
| spsurya | meeting time BRB | 07:29 |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla-kubernetes master: Deployment Guide Documentation https://review.openstack.org/447356 | 07:30 |
| *** britthouser has joined #openstack-kolla | 07:31 | |
| *** britthouser7 has quit IRC | 07:31 | |
| *** zhurong has joined #openstack-kolla | 07:34 | |
| *** tovin07 has quit IRC | 07:45 | |
| hrw | morning | 07:54 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: make yum fail on missing packages https://review.openstack.org/449086 | 07:56 |
| hrw | rebased on master | 07:57 |
| bjolo | morning | 07:57 |
| spsurya | hrw: bjolo morning | 07:58 |
| spsurya | hrw: https://review.openstack.org/#/c/440634/ ask some core for this review | 07:59 |
| spsurya | hrw: as i am not sure about this | 07:59 |
| hrw | spsurya: this one is kind of problematic to me. | 07:59 |
| spsurya | hrw: yes | 08:00 |
| hrw | spsurya: I am sure that previous state was wrong (ceph-fuse in ceph-base) but is my suggestion right? do not know | 08:00 |
| *** Mech422 has quit IRC | 08:01 | |
| spsurya | hrw: not so sure :( | 08:01 |
| hrw | gnocchi will get one patch later today. but have to check some logs first | 08:02 |
| spsurya | hrw: ok | 08:03 |
| *** shardy has joined #openstack-kolla | 08:03 | |
| hrw | gnocchi-base/source takes huge amount of time on !x86-64 | 08:05 |
| *** nathharp has joined #openstack-kolla | 08:07 | |
| *** gfidente has joined #openstack-kolla | 08:08 | |
| spsurya | hrw: if we see the existing x86 executables run with no compatibility or performance penalties | 08:09 |
| spsurya | thats include in packages itself | 08:10 |
| duonghq | I'm sorry but who is "Marcin Juszkiewicz"? | 08:10 |
| duonghq | in irc? | 08:10 |
| spsurya | hrw is | 08:10 |
| spsurya | AFAIK | 08:10 |
| spsurya | hrw: am i right ? | 08:11 |
| hrw | duonghq: me | 08:11 |
| duonghq | hrw, so I remember correctly, | 08:12 |
| duonghq | about this ps: https://review.openstack.org/#/c/449086/ | 08:12 |
| hrw | :) | 08:12 |
| hrw | duonghq: yum one, yes? | 08:12 |
| duonghq | you mean if we miss some package on x86 arch, it still fine? | 08:12 |
| hrw | duonghq: no, I mean that on x86 there are no missing ones | 08:12 |
| hrw | duonghq: those missing ones got cought and fixes got merged last week | 08:13 |
| duonghq | hrw, technically, nothing is impossible, | 08:13 |
| hrw | duonghq: sure. | 08:13 |
| duonghq | btw, you got it much more time in non-x86 arch? | 08:13 |
| *** dixiaoli has quit IRC | 08:13 | |
| hrw | duonghq: huge amount of times | 08:13 |
| duonghq | can you explain the reason? | 08:14 |
| hrw | duonghq: most of my patches are "on aarch64 or ppc64le this package is missing" | 08:14 |
| duonghq | hrw, due to fail to connect to repository? or what? | 08:14 |
| hrw | duonghq: for centos/x86-64 kolla is using several external repositories. None of them provide packages for aarch64 nor ppc64le | 08:14 |
| hrw | duonghq: so for example no grafana, no fluentd | 08:15 |
| duonghq | so, in poor connection env, there is possible that I cannot contact to any repository | 08:15 |
| duonghq | i.e. I face same issue as your | 08:15 |
| hrw | duonghq: sure, but then your build fail on "yum: cannot connect to repo" | 08:15 |
| hrw | duonghq: where on !x86-64 I do not even use those repos | 08:16 |
| *** adrian_otto has quit IRC | 08:16 | |
| duonghq | it do not output you cannot connect to these repos | 08:16 |
| duonghq | interesting | 08:17 |
| hrw | duonghq: it does. I had several CI failures last week due to grafana repo being nonaccessible | 08:17 |
| duonghq | wait | 08:17 |
| hrw | and CI runs on x86-64 | 08:17 |
| duonghq | without your patch, it still fails or not? | 08:18 |
| duonghq | sorry, but I lost some thought | 08:18 |
| hrw | it fails without | 08:18 |
| *** dixiaoli has joined #openstack-kolla | 08:18 | |
| duonghq | so, with your patch, it still fails? | 08:19 |
| hrw | with or without it fails when repo is unaccessible | 08:19 |
| hrw | because it happens before yum starts to check what it has to download/install | 08:19 |
| duonghq | ah, but w/o it, it does not fail if the repo still accessible but not contain the package? | 08:20 |
| *** pbandark has joined #openstack-kolla | 08:20 | |
| hrw | duonghq: other example: I do build of aodh-base for centos/binary. There is no openstack-aodh-common package for aarch64 so without my patch I get aodh-base image which lacks aodh. with my patch I get build failure | 08:20 |
| hrw | duonghq: if repo is not accessible then my patch does not change anything | 08:20 |
| spsurya | duonghq: +1 | 08:21 |
| spsurya | without it should not be fail | 08:21 |
| hrw | sorry, I got lost ;d | 08:21 |
| hrw | 10:20 < duonghq> ah, but w/o it, it does not fail if the repo still accessible but not contain the package? | 08:21 |
| hrw | it not fail | 08:21 |
| hrw | just be useless | 08:21 |
| duonghq | got it | 08:22 |
| duonghq | interesting case :) | 08:22 |
| hrw | it may fail if Dockerfile.j2 plays with files from package and they are not present but that's only some images | 08:22 |
| duonghq | thank for explain it, hrw | 08:22 |
| hrw | duonghq: yw | 08:22 |
| hrw | duonghq: can you mark it W+1 then? | 08:22 |
| duonghq | hrw, gate is broking again :( | 08:23 |
| *** mgoddard has joined #openstack-kolla | 08:23 | |
| spsurya | hrw: it may fail if Dockerfile.j2 plays with files from package and they are not present but that's only some images | 08:23 |
| spsurya | got you hrw | 08:23 |
| hrw | spsurya: image isntalls package. package contains /usr/share/package/package/py and Dockerfile.j2 seds it. no file == failure | 08:24 |
| * duonghq tire with gate :( | 08:24 | |
| hrw | duonghq: I meant +1 for workflow :P | 08:24 |
| duonghq | hrw, I know | 08:25 |
| hrw | :) | 08:25 |
| duonghq | but I need all gates are green :( | 08:25 |
| hrw | duonghq: gate... I agree that it is pain | 08:25 |
| spsurya | hrw: roger | 08:25 |
| duonghq | wish that I can merge gate result from different run | 08:25 |
| duonghq | and you'll have all green gate | 08:25 |
| hrw | but that would be cheating | 08:26 |
| *** matrohon has joined #openstack-kolla | 08:27 | |
| duonghq | not sure, I don't think union operator cheat anybody | 08:27 |
| duonghq | we have some code | 08:27 |
| duonghq | *same | 08:27 |
| duonghq | and have gate green for x% | 08:27 |
| duonghq | for 12 long running gate | 08:28 |
| *** matrohon_ has joined #openstack-kolla | 08:28 | |
| hrw | duonghq: sure. this time. | 08:29 |
| duonghq | we still recheck again until gate is green, so if union is a cheat, recheck is equivalent | 08:29 |
| hrw | duonghq: but patch1 can be right on d.m.y and wrong day later because patch2 got merged in meantime | 08:29 |
| duonghq | give gate is independent on others | 08:29 |
| duonghq | let me rephrase, given same code, no rebase, no edit, .... | 08:30 |
| *** gkadam is now known as gkadam-mtg | 08:30 | |
| duonghq | it should be consistence from time to time | 08:30 |
| duonghq | but it doesn't | 08:30 |
| duonghq | so we recheck | 08:30 |
| hrw | yep | 08:30 |
| duonghq | and hoping for it go green with some chance | 08:30 |
| duonghq | it's equivalent of trust to merging all test result from multiple run | 08:31 |
| duonghq | given exactly same code | 08:31 |
| *** matrohon has quit IRC | 08:31 | |
| *** yuanying has quit IRC | 08:31 | |
| spsurya | agreed with duonghq | 08:32 |
| duonghq | and bad news is your ps get red on deploy-ubuntu-source-ubuntu-xenial | 08:32 |
| *** gkadam_ has joined #openstack-kolla | 08:32 | |
| duonghq | and we need waiting half hour before next recheck | 08:33 |
| duonghq | hrw, something wrong with haproxy log in gate of your ps | 08:35 |
| duonghq | not good | 08:35 |
| *** gkadam-mtg has quit IRC | 08:35 | |
| hrw | duonghq: in which patch? | 08:35 |
| duonghq | the yum patch | 08:35 |
| duonghq | hrw, here http://logs.openstack.org/86/449086/5/check/gate-kolla-dsvm-deploy-ubuntu-source-ubuntu-xenial-nv/a774ee1/ | 08:36 |
| duonghq | doesn't related to your change | 08:36 |
| hrw | INFO:kolla.image.build.haproxy:adduser: The user `haproxy' already exists. Exiting. | 08:36 |
| hrw | that's package bug | 08:37 |
| hrw | but image got built anyway | 08:37 |
| duonghq | hrw, http://logs.openstack.org/86/449086/5/check/gate-kolla-dsvm-deploy-ubuntu-source-ubuntu-xenial-nv/a774ee1/console.html#_2017-03-29_08_14_57_188159 | 08:38 |
| duonghq | this failed | 08:38 |
| hrw | ah | 08:38 |
| duonghq | hrw, you let all of your ps related to previous work, so nobody can get anything merge soon :+ | 08:44 |
| duonghq | :( | 08:44 |
| hrw | duonghq: they use base_arch variable which is implemented in 1st patch | 08:46 |
| *** dixiaoli_ has joined #openstack-kolla | 08:46 | |
| duonghq | hrw, understand :( | 08:47 |
| duonghq | can you raise it in today meeting, | 08:47 |
| hrw | yep | 08:47 |
| duonghq | nice | 08:47 |
| *** dixiaoli has quit IRC | 08:48 | |
| hrw | addd to agenda | 08:49 |
| spsurya | duonghq: In your opinion | 08:52 |
| spsurya | what all left for this https://review.openstack.org/#/c/425446/ | 08:53 |
| spsurya | I guess we can complete this in pike right ? | 08:53 |
| spsurya | duonghq: it would be great if you will provide your thought on this | 08:54 |
| *** yangyapeng has joined #openstack-kolla | 08:57 | |
| hrw | duonghq: will it help if I split https://review.openstack.org/#/c/430940/ (1st patch in series) into two parts? One will just add base_arch variable into code. So nothing will change for anyone. but it will be simple patch which enable several patches to mergepath. | 08:57 |
| openstackgerrit | Bertrand Lallau proposed openstack/kolla-ansible master: Remove ml2_conf.ini file from some Neutron agents https://review.openstack.org/447992 | 08:57 |
| hrw | duonghq: and no other architectures added in it | 08:58 |
| duonghq | hrw, should bring the idea to the meeting, but it's nice IMO | 09:00 |
| duonghq | spsurya, gate green, so it should be good, even we didn't test it well | 09:00 |
| *** yangyape_ has joined #openstack-kolla | 09:01 | |
| duonghq | we must revised it latter | 09:01 |
| duonghq | *revise | 09:01 |
| duonghq | spsurya, I'm not sure, need more dicussion | 09:02 |
| spsurya | duonghq: i thing we should mark workflow -1 or commit tile as WIP so that we can finish testing upto a lavel | 09:02 |
| hrw | duonghq: https://paste.fedoraproject.org/paste/WuvJvmJCSUaFtx2b8OFJkF5M1UNdIGYhyRLivL9gydE=/ is whole base_arch patch | 09:02 |
| spsurya | duonghq: then we can thing about its review and merging | 09:03 |
| spsurya | as this is essential one | 09:03 |
| *** yangyapeng has quit IRC | 09:04 | |
| duonghq | spsurya, we should bring it to meeting | 09:04 |
| *** egonzalez has joined #openstack-kolla | 09:04 | |
| spsurya | duonghq: we already did that 2 times IIRC | 09:05 |
| spsurya | :) | 09:05 |
| duonghq | spsurya, sorry, my thought get jamming now | 09:05 |
| duonghq | :( | 09:05 |
| duonghq | spsurya, yes, and many CRs agree that it can be merged | 09:05 |
| spsurya | may be egonzalez can confirm that | 09:06 |
| *** manheim has joined #openstack-kolla | 09:06 | |
| spsurya | duonghq: then need review | 09:06 |
| egonzalez | spsurya about? | 09:06 |
| spsurya | https://review.openstack.org/#/c/425446/ egonzalez about this one | 09:06 |
| *** manheim has quit IRC | 09:07 | |
| *** manheim has joined #openstack-kolla | 09:07 | |
| spsurya | egonzalez: do we need to test it up to a level | 09:07 |
| duonghq | morning egonzalez | 09:08 |
| spsurya | egonzalez: at least the basic one which is mentioned https://etherpad.openstack.org/p/ks-rolling-upgrade-test | 09:08 |
| egonzalez | spsurya yep, there was a time since i tested the last time, some things may changed in keystone side | 09:09 |
| hrw | egonzalez: https://paste.fedoraproject.org/paste/WuvJvmJCSUaFtx2b8OFJkF5M1UNdIGYhyRLivL9gydE=/ - would you +2 it? | 09:09 |
| egonzalez | hrw wrong link tho | 09:10 |
| hrw | egonzalez: it's raw patch so far | 09:10 |
| spsurya | egonzalez: ok | 09:11 |
| hrw | egonzalez: I am thinking of doing my !x86-64 patchset in other way. to make it easier to merge | 09:11 |
| hrw | egonzalez: base_arch change does not change anything but allows to get several of my patches into own patchsets making main one smaller | 09:12 |
| spsurya | egonzalez: i will check and will comment on the patch set, if any changes has been done from keystone side | 09:12 |
| egonzalez | hrw i think that is the good way, otherwise changes will be failing with merge conflict forever. small non breaking changes is the good way imo | 09:13 |
| hrw | :) | 09:13 |
| hrw | will then sent it right now and then rebase on top | 09:13 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: add base_arch variable for future non-x86 work https://review.openstack.org/451268 | 09:18 |
| hrw | egonzalez, duonghq: can you review that and +2? | 09:18 |
| *** gkadam_ has quit IRC | 09:20 | |
| *** rmart04 has joined #openstack-kolla | 09:21 | |
| egonzalez | hrw not sure if some tests for base arch should be included in test_build, not really familiar with code coverage | 09:22 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: add base_arch variable for future non-x86 work https://review.openstack.org/451268 | 09:25 |
| hrw | found bug ;( | 09:25 |
| hrw | solved | 09:25 |
| pomac | anyone that can help me wrap my head around ansible - i've enable debug and *_KEEP_REMOTE_* environmental variables, all of this leads me to belive that a error is local, in the generation of ansiballz or so - https://pastebin.com/yx81BAkU <- execution trace - the result doesn't seem to ever be stored on the remote host and from what i gather all data is available | 09:26 |
| pomac | The failure is: "msg": "'dict object' has no attribute u'ansible_bond0'" | 09:26 |
| egonzalez | pomac if you do ansible -m setup -i inventory host ? is bond0 in the list of ansible interfaces? | 09:27 |
| pomac | In the debug promt: p vars['hostvars'][str(host)]['ansible_bond0'] works | 09:27 |
| pomac | yes | 09:27 |
| pomac | this is only when it seems to generate the nova config - the merge pass | 09:27 |
| pomac | egonzalez: trying to figure out where in ansible this happens is.... hard... =) | 09:28 |
| pomac | p vars['hostvars'][str(host)]['ansible_bond0']['ipv4']['address'] also works | 09:29 |
| pomac | (lunch, afk) | 09:29 |
| *** sambetts|afk is now known as sambetts | 09:30 | |
| *** gkadam_ has joined #openstack-kolla | 09:31 | |
| *** gkadam_ is now known as gkadam | 09:35 | |
| *** cu5 has joined #openstack-kolla | 09:35 | |
| openstackgerrit | shaofeng cheng proposed openstack/kolla-ansible master: Add networking with vmware vsphere https://review.openstack.org/451289 | 09:36 |
| *** yuanying has joined #openstack-kolla | 09:39 | |
| *** manheim has quit IRC | 09:41 | |
| *** yuanying has quit IRC | 09:44 | |
| *** caoyuan has quit IRC | 09:45 | |
| *** yuanying has joined #openstack-kolla | 09:47 | |
| *** duonghq has quit IRC | 09:48 | |
| hawi | hi. what is needed in order to get nice grafana dashboard about kolla installation? i'm just trying to figure out, what to use for it, collectd, influxdb, elasticsearch, telegraf? | 09:52 |
| *** yingjun has quit IRC | 09:54 | |
| openstackgerrit | Surya Prakash (spsurya) proposed openstack/kolla-kubernetes master: py34 not in use for the deliverable https://review.openstack.org/451300 | 09:56 |
| *** matrohon_ has quit IRC | 09:57 | |
| *** yangyape_ has quit IRC | 09:59 | |
| *** manheim has joined #openstack-kolla | 10:00 | |
| *** yangyapeng has joined #openstack-kolla | 10:00 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: debian: enable all images enabled for Ubuntu https://review.openstack.org/432787 | 10:07 |
| hrw | another change extracted from huge patchset into separate one | 10:07 |
| egonzalez | hrw added a comment https://review.openstack.org/#/c/449086/ | 10:15 |
| egonzalez | features cannot be backported to stable branchs, and partially-implements in commit is saying that is a feature | 10:16 |
| hrw | ok | 10:18 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: make yum fail on missing packages https://review.openstack.org/449086 | 10:18 |
| hrw | egonzalez: dropped it then | 10:18 |
| * hrw -> splitting patchset into parts | 10:20 | |
| egonzalez | hrw thanks, atm were are applying for stable policy, don't want to risk that | 10:20 |
| hrw | sure | 10:20 |
| hrw | egonzalez: I care more about getting patch in that will it be linked to blueprint ;d | 10:20 |
| *** tovin07_ has quit IRC | 10:20 | |
| hrw | s/that/than | 10:20 |
| *** cuongnv has quit IRC | 10:21 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: ironic-pxe: handle non-x86 architectures https://review.openstack.org/434817 | 10:26 |
| openstackgerrit | Surya Prakash (spsurya) proposed openstack/kolla-kubernetes master: Deployment Guide Documentation https://review.openstack.org/447356 | 10:27 |
| *** pbourke has quit IRC | 10:27 | |
| *** pbourke has joined #openstack-kolla | 10:28 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: kubernetes: disable for architectures other than x86-64 https://review.openstack.org/445638 | 10:29 |
| *** caowei has quit IRC | 10:30 | |
| pbourke | egonzalez: Daviey Jeffrey4l zhubingbing: hi can some of you please review https://review.openstack.org/#/c/438030/ | 10:30 |
| pbourke | thanks | 10:30 |
| zhubingbing | ok | 10:30 |
| *** MarginHu has quit IRC | 10:32 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: use mariadb-devel for CentOS and !x86-64 https://review.openstack.org/444832 | 10:37 |
| hrw | pbourke: what do you think about https://review.openstack.org/451268 one? | 10:38 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: nova-compute: handle rtslib(-fb) for debian-binary builds too https://review.openstack.org/435941 | 10:39 |
| pbourke | hrw: "...requires checking again and again after each change/merge/refresh" | 10:40 |
| pbourke | hrw: what do you mean? | 10:40 |
| hrw | pbourke: each merge conflict (~every second day) I have to push whole patchset and then anyone who did review gets flooded with 'please review patch which did not changed since last time but got reuploaded' | 10:42 |
| hrw | pbourke: and patchset grows and grows | 10:42 |
| pbourke | so how does this solve that? | 10:43 |
| hrw | pbourke: with base_arch in master most of patches are separate from each other | 10:43 |
| pbourke | ok | 10:43 |
| hrw | pbourke: they do not need non-x86 support merged to work fine on x86-64 | 10:44 |
| hrw | and once non-x86 get merged they magically work on non-x86 too | 10:44 |
| pbourke | hrw: one small comment on the usage msg | 10:45 |
| hrw | pbourke: I am open to comments | 10:45 |
| hrw | will take care | 10:46 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: ceph: move ceph-fuse package to cephfs-fuse image https://review.openstack.org/440634 | 10:46 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: add base_arch variable for future non-x86 work https://review.openstack.org/451268 | 10:49 |
| *** egonzalez has quit IRC | 10:50 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: openstack-base: Percona-Server is x86-64 only https://review.openstack.org/449965 | 10:52 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: nova-compute: enable Debian/source build https://review.openstack.org/450805 | 10:54 |
| hrw | ok, now most of patches should be not related to each other | 10:55 |
| manheim | hi, i have a question regarding multinode and docker registry https://docs.openstack.org/developer/kolla/newton/multinode.html | 10:56 |
| manheim | the registry can be on any host on my lan, right? | 10:56 |
| manheim | then on the deployer node I specify the IP of my registry | 10:56 |
| *** zhubingbing has quit IRC | 10:56 | |
| *** qiliang27 has quit IRC | 10:58 | |
| *** qiliang27 has joined #openstack-kolla | 10:58 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: mariadb: handle lack of external repos on non x86 https://review.openstack.org/444832 | 11:03 |
| *** yangyapeng has quit IRC | 11:05 | |
| *** matrohon_ has joined #openstack-kolla | 11:07 | |
| pbourke | hrw: just fyi, there's really no need to track your different patchsets in the commit msg | 11:07 |
| hrw | pbourke: you mean my comment in 444832 one? | 11:08 |
| hrw | pbourke: I prefer to mention it to not loose track on my own too | 11:09 |
| *** nathharp has quit IRC | 11:09 | |
| hrw | 13:09 hrw@gossamer:kolla$ git branch |grep to-merge|wc -l | 11:09 |
| hrw | 19 | 11:09 |
| * hrw off for ~1h | 11:11 | |
| *** thomas_oneill is now known as thomas-oneill | 11:16 | |
| *** zhurong has quit IRC | 11:34 | |
| *** matrohon_ has quit IRC | 11:37 | |
| *** rwallner has joined #openstack-kolla | 11:40 | |
| *** gkadam is now known as gkadam-afk | 11:47 | |
| *** blallau has joined #openstack-kolla | 11:48 | |
| *** dixiaoli_ has quit IRC | 11:49 | |
| *** zhubingbing has joined #openstack-kolla | 11:49 | |
| *** dixiaoli has joined #openstack-kolla | 11:50 | |
| *** pbandark is now known as pbandark`brb | 11:52 | |
| *** rhallisey has joined #openstack-kolla | 11:54 | |
| *** dixiaoli has quit IRC | 11:54 | |
| openstackgerrit | Mick Thompson proposed openstack/kolla-ansible master: Update keystone haproxy config to balance based on source ip https://review.openstack.org/451358 | 11:55 |
| *** athomas has quit IRC | 11:57 | |
| *** Pavo has joined #openstack-kolla | 11:58 | |
| *** athomas has joined #openstack-kolla | 12:05 | |
| *** cu5 has quit IRC | 12:07 | |
| *** egonzalez has joined #openstack-kolla | 12:07 | |
| sdake | Jeffrey4l around? | 12:09 |
| mgoddard | hi pbourke, would you be able to re-review and (ideally) restore your +2 on https://review.openstack.org/#/c/436462/? | 12:09 |
| sdake | morning flks | 12:09 |
| sdake | sup mgoddard | 12:09 |
| mgoddard | hey sdake | 12:10 |
| sdake | mgoddard there is a patch in the rview queue related to ironic binary | 12:10 |
| sdake | and sudoers file | 12:10 |
| egonzalez | morning sdake | 12:10 |
| sdake | sup egonzalez | 12:10 |
| mgoddard | sdake: correct. would be nice to move it along | 12:10 |
| sdake | mgoddard i thought you fixed all that? | 12:11 |
| sdake | mgoddard have a link? | 12:11 |
| mgoddard | sdake: assume you're referring to https://review.openstack.org/441605 and https://review.openstack.org/445587? | 12:11 |
| sdake | mgoddard no something else | 12:12 |
| sdake | mgoddard lemme wake up alittle its 5am - and i'll find a link if possible | 12:13 |
| mgoddard | sdake: sure, let me know | 12:13 |
| *** gkadam-afk is now known as gkadam | 12:14 | |
| mgoddard | sdake: you referring to this: https://review.openstack.org/#/c/448977? | 12:14 |
| *** signed8bit_Zzz is now known as signed8b_ | 12:14 | |
| sdake | no | 12:16 |
| *** kwazar_ is now known as kwazar | 12:22 | |
| *** schwicht has joined #openstack-kolla | 12:29 | |
| sdake | mgoddard https://review.openstack.org/#/c/450274/ | 12:31 |
| hrw | re | 12:33 |
| sdake | mgoddard does that need a backport? | 12:33 |
| *** schwicht has quit IRC | 12:35 | |
| *** bjolo has quit IRC | 12:35 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: mariadb: handle lack of external repos on non x86 https://review.openstack.org/444832 | 12:37 |
| *** admin0 has joined #openstack-kolla | 12:37 | |
| *** schwicht has joined #openstack-kolla | 12:38 | |
| mnaser | pbourke could you remove your -1 as it was shown that this was an actual cherry picked change | 12:39 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: ironic-pxe: handle non-x86 architectures https://review.openstack.org/434817 | 12:39 |
| egonzalez | mliima around | 12:39 |
| *** krtaylor has quit IRC | 12:40 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: kubernetes: disable for architectures other than x86-64 https://review.openstack.org/445638 | 12:40 |
| hrw | mliima: thx for Depends-On reminder | 12:41 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: openstack-base: Percona-Server is x86-64 only https://review.openstack.org/449965 | 12:41 |
| hrw | can someone upvote https://review.openstack.org/#/c/449086/ (yum)? | 12:43 |
| openstackgerrit | Bertrand Lallau proposed openstack/kolla master: Remove Ubuntu Trove postinst script hack https://review.openstack.org/451378 | 12:44 |
| *** shashank_t_ has joined #openstack-kolla | 12:45 | |
| *** dave-mccowan has joined #openstack-kolla | 12:46 | |
| Jeffrey4l | sdake, yep. | 12:47 |
| *** rwallner has quit IRC | 12:47 | |
| sdake | Jeffrey4l did inc0 talk to yo uabout todays's meeting? | 12:47 |
| *** signed8b_ is now known as signed8bit_Zzz | 12:47 | |
| *** rwellum has joined #openstack-kolla | 12:48 | |
| *** Deys has joined #openstack-kolla | 12:48 | |
| Jeffrey4l | sdake, yep. i will hold today's meeting. | 12:48 |
| *** lamt has joined #openstack-kolla | 12:48 | |
| *** matrohon_ has joined #openstack-kolla | 12:48 | |
| *** shashank_t_ has quit IRC | 12:50 | |
| *** lamt has quit IRC | 12:52 | |
| mliima | hi egonzalez | 12:52 |
| sbezverk | good morning, new kube 1.6.0 bits broke our gate. I filed the issue with kubernetes developers: https://github.com/kubernetes/kubernetes/issues/43805 | 12:52 |
| mliima | hrw, :) | 12:53 |
| *** gkadam has quit IRC | 12:53 | |
| sdake | sbezverk not surprising | 12:54 |
| sdake | sbezverk a million things could have changed | 12:54 |
| sdake | sbezverk which versoin of docker-engine is being used? 12.6? | 12:55 |
| sdake | I am pretty sure Iwant with 1.6.0-rc1 with kubeadm and it did deploy | 12:55 |
| *** schwicht has quit IRC | 12:55 | |
| hrw | https://review.openstack.org/#/c/451268/ (base_arch variable) - any comments (or upvotes)? | 12:56 |
| *** goldyfruit has joined #openstack-kolla | 12:59 | |
| *** lamt has joined #openstack-kolla | 13:00 | |
| *** bswartz has left #openstack-kolla | 13:00 | |
| openstackgerrit | Merged openstack/kolla master: Make keystone_user module fully shade https://review.openstack.org/448014 | 13:04 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Fixing 1.6.0 kubeadm init getting stuck https://review.openstack.org/451391 | 13:10 |
| *** mattmceuen has joined #openstack-kolla | 13:10 | |
| *** rwallner has joined #openstack-kolla | 13:12 | |
| *** matrohon_ has quit IRC | 13:13 | |
| *** JoseMello has joined #openstack-kolla | 13:14 | |
| *** Deys has quit IRC | 13:14 | |
| rwellum | sdake: sbezverk argh I was seeing that last night following the guide - I was sure I was doing something wrong... | 13:16 |
| *** lamt has quit IRC | 13:17 | |
| mnaser | lol | 13:18 |
| mnaser | bluebox cloud has a host running on 169.254.169.10 | 13:18 |
| *** schwicht has joined #openstack-kolla | 13:18 | |
| mnaser | so prechecks cause job failures | 13:18 |
| sbezverk | rwellum: nope, and that fix is not enough, other things got broken too | 13:18 |
| mnaser | http://logs.openstack.org/03/450203/2/check/gate-kolla-dsvm-deploy-centos-binary-centos-7-nv/f917f47/console.html | 13:18 |
| mnaser | fun times | 13:19 |
| *** erlon has joined #openstack-kolla | 13:20 | |
| *** rwallner has quit IRC | 13:21 | |
| rwellum | sbezverk: 😢 | 13:22 |
| *** rwallner has joined #openstack-kolla | 13:23 | |
| *** zhurong has joined #openstack-kolla | 13:23 | |
| hawi | is this something usua: The content of source file(/var/lib/kolla/config_files/galera.cnf) and dest file(/etc/mysql/my.cnf) are not equal | 13:25 |
| hawi | i cant start galera any more | 13:25 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Fixing 1.6.0 kubeadm init getting stuck https://review.openstack.org/451391 | 13:26 |
| *** Deys has joined #openstack-kolla | 13:28 | |
| *** signed8bit_Zzz is now known as signed8b_ | 13:28 | |
| *** jmontleon has quit IRC | 13:29 | |
| *** jmontleon has joined #openstack-kolla | 13:29 | |
| *** mliima_ has joined #openstack-kolla | 13:31 | |
| sdake | rwellum note people yesterday evening reported the guide got them operational | 13:31 |
| *** esharao has joined #openstack-kolla | 13:31 | |
| sdake | rwellum although it was about10pm MST so i hit the rack | 13:31 |
| openstackgerrit | Mauricio Lima proposed openstack/kolla-ansible master: Pg_num should be configurable https://review.openstack.org/450797 | 13:31 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla master: Updated from global requirements https://review.openstack.org/451016 | 13:33 |
| *** eanylin has quit IRC | 13:34 | |
| *** mliima has quit IRC | 13:34 | |
| *** zhubingbing has quit IRC | 13:37 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible master: Fix ceph log location https://review.openstack.org/451407 | 13:37 |
| *** jmontleon has quit IRC | 13:38 | |
| *** jaosorior has quit IRC | 13:38 | |
| *** jtriley has joined #openstack-kolla | 13:39 | |
| sdake | sbezverk looking in the yum repos, i dont see a kubernetes 1.6.0 | 13:39 |
| *** dmsimard is now known as dmsimard|afk | 13:39 | |
| sdake | sbezverk although clearly the gate is using it | 13:39 |
| *** jmontleon has joined #openstack-kolla | 13:41 | |
| sdake | there are two version in the repo rwellum | 13:41 |
| sdake | rwellum you can install kubernetes and kubelet 1.5.4 if you want to follow the instructions | 13:42 |
| rwellum | sdake: not blaming the guide -it's looking really good. In fact I thought it was my VM acting up. | 13:43 |
| sdake | sbezverk re kubeadm - did you try the version flag to install kubernetes-1.6.0 | 13:43 |
| *** jaosorior has joined #openstack-kolla | 13:43 | |
| sdake | rwellum not saying you did blame guide, only trying to help you get up and running :) | 13:43 |
| rwellum | sdake: maybe stupid question, why are we not using tools/setup_kubernetes.sh ? | 13:43 |
| sdake | sbezverk i suspect kubeadm will work better with teh proper version flag | 13:44 |
| *** mbruzek has joined #openstack-kolla | 13:44 | |
| *** mbruzek has joined #openstack-kolla | 13:44 | |
| sdake | rwellum we possibly could - I personally want to understand the workflow rather then some magic script | 13:44 |
| *** mbruzek has quit IRC | 13:44 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Fixing 1.6.0 kubeadm init getting stuck https://review.openstack.org/451391 | 13:44 |
| sdake | rwellum setup_kubernetes.sh is designed to only run in the gate, not as a general purpose guide | 13:44 |
| rwellum | Well we have a few magic scripts - you wrote one to clean up k8s. | 13:44 |
| rwellum | Oh IC. | 13:45 |
| rwellum | Now that would be a great comment at the top of the script... | 13:45 |
| sdake | rwellum ya that magic cleanup-k8s script is a little frightening :) | 13:45 |
| sdake | rwellum actually vhosakot wrote most of that | 13:45 |
| sdake | i debated putting it in the review at all | 13:46 |
| sdake | i may end up taking it out | 13:46 |
| esharao | GM guys | 13:47 |
| sdake | sup esharao | 13:47 |
| rwellum | sdake: so in the guide we call out versions at the top - maybe we need to add those versions to the steps to install? We can always update the versions when tested. | 13:47 |
| esharao | was retrying the guide.. with the new placement stuff called out by eanylin | 13:47 |
| esharao | still my nova compute keeps on crashing but everything else is running.. | 13:48 |
| esharao | managed to get my libvirt up once i removed profile from apparmour (maybe thats something to add for folks who use ubuntu) | 13:48 |
| sdake | esharao i'm not sure eanylin tested it - he indicated he thought it was correct but was clear he wasn't sure last noight on irc | 13:49 |
| *** jmontleon has quit IRC | 13:49 | |
| *** dciabrin has quit IRC | 13:49 | |
| openstackgerrit | Mauricio Lima proposed openstack/kolla-ansible master: Pg_num should be configurable https://review.openstack.org/450797 | 13:49 |
| esharao | oh.. was worth a shot .. :) easy to helm delete and helm install.. | 13:49 |
| sdake | rwellum ya - ther eis a flag to kubeadm to run a specific version | 13:49 |
| *** jmontleon has joined #openstack-kolla | 13:49 | |
| sdake | esharao he did mention it had to do with the order in which they were launched | 13:49 |
| sdake | esharao and I think thesame | 13:50 |
| mgoddard | sdake: thanks for the link, sorry for the slow reply. I'll comment on the that review - I'm not sure it's quite right. | 13:50 |
| mgoddard | sdake: As for a backport, looks like it (or something similar) needs to go into stable/ocata but the issue should not be present in 4.0.0 thankfully. | 13:50 |
| sdake | mgoddard thanks - so do we need to revert thta change? | 13:50 |
| sdake | mgoddard you had indicated you had ironic working for binary for 4.0.0 | 13:50 |
| *** rwallner has quit IRC | 13:51 | |
| sdake | mgoddard or did you ahve it working for some other version of the containers? | 13:52 |
| esharao | sdake.. one question.. even though all other images have tag 4.0.0, my kolla-toolbox and haproxy have 3.0.2... Is that how its supposed to be? | 13:52 |
| mgoddard | sdake: I don't believe it warrants a revert - it should work. It just introduces an additional sudoers entry for the binary build that is unnecessary. | 13:52 |
| sdake | mgoddard trying to understand the impact for 4.0.0 images specifically | 13:52 |
| sdake | esharao haproxy should probably have a 4.0.0 tag | 13:52 |
| sdake | esharao toolbox shoudl work fine with a 4.0.0 tag - although I haven't personally tested it | 13:53 |
| mgoddard | sdake: 4.0.0 should be fine - it did not include the iscsi_tcp modprobe that requires this sudoers entry | 13:53 |
| sdake | mgoddard cool so no backport needed | 13:54 |
| mgoddard | sdake: I haven't specifically tested ironic binary at 4.0.0 | 13:54 |
| sdake | is this iscsp_tcp modprobe a new feature in pike? | 13:54 |
| *** jrist has quit IRC | 13:55 | |
| sdake | mgoddard ack - you tested 3.0.2 binary then? | 13:55 |
| *** eanylin has joined #openstack-kolla | 13:55 | |
| mgoddard | sdake: a backport is required because the iscsi_tcp modprobe was committed to stable/ocata after 4.0.0 | 13:55 |
| sdake | mgoddard thats a feature - you ha eto be kidding me! | 13:55 |
| mgoddard | sdake: IIRC I was testing ironic inspector | 13:55 |
| sdake | mgoddard fwiw, not lecturing you - jsut surprised there is a feature going into stable/ocata | 13:56 |
| esharao | sdake how do i force it since in my cloud.yaml tag is for 4.0.0 | 13:56 |
| mgoddard | sdake: it's a bug fix - iscsi-based ironic deployments fail without it | 13:56 |
| sdake | esharao there is an order of precedent in cloud.yaml | 13:57 |
| mgoddard | sdake: although I wasn't involved in the fix | 13:57 |
| sdake | mgoddard ya - its not your fault - was just venting outloud sorry | 13:57 |
| sdake | esharao the way the orderof precedent works is as follows | 13:57 |
| mgoddard | sdake: np :) | 13:57 |
| sdake | kolla.all.* is applied -> then kolla.service.* is applied | 13:57 |
| *** jrist has joined #openstack-kolla | 13:57 | |
| sdake | esharao this precedence mens if you put the version in all, it will aply to all entries | 13:58 |
| esharao | yes.. thats what i have | 13:58 |
| sdake | if you want to override the image tag for a sub-service, you have to do so | 13:58 |
| *** srwilkers has joined #openstack-kolla | 13:58 | |
| sdake | morning srwilkers | 13:58 |
| esharao | http://paste.openstack.org/show/604662/ | 13:58 |
| srwilkers | morning sdake | 13:58 |
| esharao | oh i do not want to.. it just happened that way.. and was curious as to why it happened | 13:59 |
| sdake | esharao i believe that should make all yoru images go to 4.0.0 | 13:59 |
| sdake | esharao line 42 is the lowest precedence | 13:59 |
| sdake | if you put an image_tag in one of the ot her services, it would take precedence over line 24 | 14:00 |
| sdake | 42 | 14:00 |
| eanylin | esharao: I tested with 4.0.0 image | 14:00 |
| sdake | esharao I should probably put that precedence explination in the docs | 14:00 |
| eanylin | based on what I am seeing, cell0 should be executed after nova-compute is stable | 14:00 |
| esharao | sdake.. that would make sense.. | 14:00 |
| sdake | eanylin so a sync point is needed? | 14:01 |
| esharao | eanylin.. i need to check as to how my tookbox took 3.0.2 | 14:01 |
| sdake | esharao with that cloud.yaml you pasted, your toolbox is 3.0.2? | 14:01 |
| eanylin | sdake: I think so | 14:01 |
| esharao | yes | 14:01 |
| esharao | let me paste output of docker images | 14:01 |
| sdake | ok - well in that case the precedence ordering is not implemented correctly in the helm chart | 14:01 |
| sdake | let me look at the code | 14:01 |
| esharao | http://paste.openstack.org/show/604663/ | 14:02 |
| eanylin | I did another test, if I try and do nova-api-create-simple-cell-job before nova-compute it up | 14:02 |
| eanylin | it will crash | 14:02 |
| *** dciabrin has joined #openstack-kolla | 14:02 | |
| eanylin | which I think makes sense since it needs to know which hypervisor can be used | 14:02 |
| *** Pavo has quit IRC | 14:04 | |
| *** zhurong_ has joined #openstack-kolla | 14:06 | |
| *** lucasxu has joined #openstack-kolla | 14:07 | |
| *** zhurong has quit IRC | 14:07 | |
| *** wirehead_ has quit IRC | 14:07 | |
| egonzalez | eanylin yeah, simple cell first sync db with cell tables and then do a discover hosts which need compute nodes lookup by nova conductor | 14:08 |
| *** wirehead_ has joined #openstack-kolla | 14:09 | |
| egonzalez | meeting in 1 or 2 hours? feeling lost after time change last week :/ | 14:09 |
| *** dciabrin has quit IRC | 14:09 | |
| sdake | egonzalez utc never changes | 14:10 |
| sdake | egonzalez 2 hours | 14:10 |
| eanylin | egonzalez: Thanks | 14:10 |
| sdake | sbezverk quick q - why are these image tags specifiable: https://github.com/openstack/kolla-kubernetes/blob/master/helm/all_values.yaml#L11-L14 | 14:10 |
| egonzalez | sdake thanks | 14:10 |
| sdake | sbezverk is it because they are in the common library? | 14:11 |
| sdake | 12-14 that is | 14:11 |
| sdake | egonzalez google recently added a way to determine times into their search tool | 14:13 |
| sdake | egonzalez you can type in questions like "what time is 16:00UTC in location" | 14:13 |
| sdake | and it should pop out the exact time | 14:14 |
| sbezverk | sdake: because these are side car images, they belong to common section.. | 14:14 |
| sdake | sbezverk right so side car images can't be overridden generally then? | 14:15 |
| sdake | sbezverk i think what would make sense is for image_tag to override sidecar image tags | 14:15 |
| sbezverk | sdake: no | 14:15 |
| sdake | sbezverk that way people don't have to specify 4 different tags to override the defautls with one tag | 14:15 |
| sdake | sbezverk not asking you to do the work - curious if that is possible | 14:16 |
| sdake | sbezverk might be a good thing for me ot try out to learn how this stuff works ;) | 14:16 |
| sbezverk | sdake: if you want to develop, then sure.. I do not see value in this.. | 14:16 |
| sdake | sbezverk atm our image_tag default is 2.0.0 haproxy tagi s 3.0.2 and toolbox tag is 3.0.2 | 14:17 |
| sdake | sbezverk its kind of wonky to ask people to specify sidecar tag #'s since they may not even know what a sidecar container is :) | 14:17 |
| sdake | tag #S/tag ids | 14:17 |
| *** zhurong_ has quit IRC | 14:18 | |
| sdake | sbezverk is it simply a matter of modifying _common_defines.yaml? | 14:18 |
| sbezverk | sdake: cannot say from the top of my head. I doubt it is a simple modification.. you will need to dig into it.. | 14:19 |
| *** rwallner has joined #openstack-kolla | 14:21 | |
| *** dciabrin has joined #openstack-kolla | 14:27 | |
| *** duonghq has joined #openstack-kolla | 14:27 | |
| duonghq | evening guys | 14:27 |
| *** shashank_t_ has joined #openstack-kolla | 14:28 | |
| *** magicboiz has joined #openstack-kolla | 14:34 | |
| *** adrian_otto has joined #openstack-kolla | 14:34 | |
| *** Pavo has joined #openstack-kolla | 14:38 | |
| *** krtaylor has joined #openstack-kolla | 14:39 | |
| *** fooliouno_ has quit IRC | 14:41 | |
| *** manheim has quit IRC | 14:44 | |
| rwellum | sbezverk: so you think the kubelet 'issue' is a new feature? Won't be fixed. | 14:44 |
| *** manheim has joined #openstack-kolla | 14:45 | |
| *** lamt has joined #openstack-kolla | 14:46 | |
| sbezverk | rwellum: not sure whether it is a new feature or not, but it seems they change defaults or added it as a mandatory config.. | 14:46 |
| duonghq | does our meeting time affect by Daylight saving time? due to its in UTC, so I think it doesn't? | 14:47 |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla-ansible master: Update keystone haproxy config to balance based on source ip https://review.openstack.org/451358 | 14:49 |
| *** matrohon_ has joined #openstack-kolla | 14:55 | |
| *** rwsu has quit IRC | 14:56 | |
| *** manheim has quit IRC | 14:56 | |
| sdake | sbezverk ok - well it may not have much value for kolla-kubernetes, but it would have value to me to learn how to write helm code | 14:58 |
| sdake | sbezverk so as long as folks don't block it - i'll do the work | 14:58 |
| sdake | sbezverk after I finish this deploy guide | 14:59 |
| sdake | duonghq UTC does not take into account DST | 14:59 |
| sdake | duonghq meeting is in 1 hour | 14:59 |
| sdake | sbezverk I think if you use --use-kubernetes-version string Choose a specific Kubernetes version for the control plane (default "stable") | 15:00 |
| sdake | you may get that value set automatically | 15:00 |
| esharao | sdake do you think its worth trying to change the toolbox to 4.0.0 in cloud yaml to see if it fixes my nova compute crashing | 15:00 |
| sdake | sbezverk i'd do this myself - however its school hour - can try in a bit | 15:00 |
| duonghq | sdake, roger | 15:00 |
| *** Deys has quit IRC | 15:00 | |
| sdake | esharao you can change it but i believe that has zero to do with nova compute crashing | 15:00 |
| esharao | oh ok :( any other pointers | 15:01 |
| sdake | esharao i'd question whether the toolbox 4.0.0 images work at all for kolla-kubernetes since nobody has tested that | 15:01 |
| *** jistr is now known as jistr|mtg | 15:01 | |
| esharao | i think eanylin has 4.0.0 | 15:01 |
| sdake | esharao the best we have so far is in the review, and eanylin seems to have sorted out the start order | 15:01 |
| sdake | eanylin mind posting your cloud.yaml? | 15:01 |
| sdake | i've gotta get my kid rolling- bbiaf :) | 15:02 |
| openstackgerrit | Merged openstack/kolla-ansible master: Improve developer documentation for Bifrost https://review.openstack.org/436462 | 15:02 |
| *** zhubingbing has joined #openstack-kolla | 15:03 | |
| eanylin | sdake: ok, actually it is similar to what we have, http://paste.openstack.org/show/604673/ | 15:03 |
| eanylin | but I need to mention that I have 2 versions of toolbox on my machine | 15:03 |
| eanylin | [root@kube03 ~]# docker images | grep toolbox | 15:04 |
| eanylin | kolla/centos-source-kolla-toolbox 4.0.0 355042fb3f2b 2 weeks ago 733 MB | 15:04 |
| eanylin | kolla/centos-source-kolla-toolbox 3.0.2 9f8d4f8130db 2 months ago 689 MB | 15:04 |
| eanylin | [root@kube03 ~]# | 15:04 |
| *** matrohon_ has quit IRC | 15:05 | |
| *** matrohon_ has joined #openstack-kolla | 15:06 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Revert "Fixes problem with kolla-kubernetes 3.0.2 images" https://review.openstack.org/444351 | 15:06 |
| sdake | eanylin which oen is running? | 15:07 |
| sbezverk | gents, you do not need to worry about toolbox, as long as it is higher than 3.0.2, it is fine.. there was no changes done to this container recently | 15:07 |
| sdake | eanylin it looks like with that config - toolbox 3.0.2 is running | 15:07 |
| sdake | esharao ^^ | 15:07 |
| eanylin | sdake: I may not have gotten the full context, what does that mean? | 15:07 |
| eanylin | sdake: Ok | 15:07 |
| sdake | sbezverk cool thanks for the confirmation - i remember something was done with toolbox - not sure which version it ultimately went into (if it went into master of kolla repos or not) | 15:08 |
| *** Pavo has quit IRC | 15:08 | |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Try the using-kubernetes-version flag to fix gate https://review.openstack.org/451451 | 15:10 |
| sbezverk | gents, could you please run this command on your kube clusters "kubeadm version" and let me know if anybody has version lower than 2074 | 15:10 |
| eanylin | sdake: Seems like its using 3.0.2 toolbox | 15:10 |
| eanylin | [root@kube03 ~]# kubectl describe pod nova-compute-dxp6p -n kolla | grep Image | 15:10 |
| eanylin | Image:docker.io/kolla/centos-source-kubernetes-entrypoint:4.0.0 | 15:10 |
| eanylin | Image ID:docker-pullable://kolla/centos-source-kubernetes-entrypoint@sha256:e01591cfdb5def77ebc0f243b4d3fe7c94c19acac7ed41a73bdbca82fbcdd312 | 15:10 |
| eanylin | Image:docker.io/kolla/centos-source-kolla-toolbox:3.0.2 | 15:10 |
| eanylin | Image ID:docker-pullable://kolla/centos-source-kolla-toolbox@sha256:fe10cce8325402a8c0a175ec88f11be13c8ff46430f6ef22a30c962c817adcbb | 15:10 |
| eanylin | Image:docker.io/kolla/centos-source-nova-compute:4.0.0 | 15:10 |
| eanylin | Image ID:docker-pullable://kolla/centos-source-nova-compute@sha256:7b6c58f8f7df7a44b732ddd9586a34fc1ece5c104c88ed3da247b4ba20737c4c | 15:10 |
| eanylin | Image:docker.io/kolla/centos-source-nova-compute:4.0.0 | 15:10 |
| eanylin | Image ID:docker-pullable://kolla/centos-source-nova-compute@sha256:7b6c58f8f7df7a44b732ddd9586a34fc1ece5c104c88ed3da247b4ba20737c4c | 15:10 |
| eanylin | [root@kube03 ~]# | 15:10 |
| sdake | yup that makes sense | 15:10 |
| eanylin | yeah | 15:10 |
| sdake | your using toolbox 3.0.2 images | 15:10 |
| *** Padouch has joined #openstack-kolla | 15:11 | |
| sdake | as toolbox is a sidecar and there is no way to globally override sidecar contianers atm | 15:11 |
| eanylin | sdake: My nova-compute been stable for past 2 hours | 15:11 |
| duonghq | Jeffrey4l, zhubingbing do you have any idea about this error: http://logs.openstack.org/16/451016/2/check/gate-kolla-dsvm-deploy-ubuntu-source-ubuntu-xenial-nv/ced7a08/console.html#_2017-03-29_14_02_31_970485 | 15:11 |
| eanylin | [root@kube03 ~]# kubectl get pods -n kolla | grep compute | 15:11 |
| eanylin | nova-compute-dxp6p 1/1 Running 2 2h | 15:11 |
| eanylin | [root@kube03 ~]# | 15:11 |
| sdake | toolbox 3.0.2 is not problematic - the biggest problem folks wiill run into atm is the fact that kubernetes 1.6.0 has been released and we are not sure if kubeadm is compatible | 15:11 |
| duonghq | I see it many times | 15:11 |
| *** manheim has joined #openstack-kolla | 15:11 | |
| sdake | eanylin nice - I actually got compute running as well | 15:11 |
| eanylin | sdake: Agree | 15:11 |
| sdake | eanylin what I wasn't able to do is create a vm - vm got stuck in BUILDING state | 15:11 |
| *** dmsimard|afk is now known as dmsimard | 15:11 | |
| esharao | hmm .. maybe i should just chuck my ubuntu and go with centos | 15:12 |
| sdake | eanylin i got compute running by mostly running the instrucitons in the review | 15:12 |
| eanylin | yeah, the cluster did come up. I still had problems with VM creations though | 15:12 |
| eanylin | yeah, I did similar too | 15:12 |
| *** sayantan_ has joined #openstack-kolla | 15:12 | |
| sdake | esharao it might make things simpler for the short term - although after we finish this guide adding ubuntu specific instructions might help :) | 15:12 |
| eanylin | sdake: Was trying to add comments along the way :) | 15:12 |
| openstackgerrit | sean mooney proposed openstack/kolla master: introduces support for the OVS DPDK dataplane https://review.openstack.org/342354 | 15:12 |
| sdake | eanylin yup eveyrone has been doing a great job on that review :) | 15:13 |
| esharao | ok.. will re-start.. that will give a test to the doc as well | 15:13 |
| rwellum | +1 to using centos for now. | 15:13 |
| esharao | talking about versions.. does the docker version matter? | 15:14 |
| *** rwallner_ has joined #openstack-kolla | 15:14 | |
| esharao | i used the 17xx CE version | 15:14 |
| esharao | but the guide says docker == 1.12.6 | 15:14 |
| eanylin | esharao: I am using 17xx CE version too | 15:15 |
| Jeffrey4l | duonghq, hrm, it is caused by a soft link. may be we should remove this feature. | 15:15 |
| rwellum | I'm a little confused with the cgroup change in kubectl if it's as per design (in which case in our document we either change input params, or point to a known earlier working release), or it's a bug that we just deal with. sbezverk suggests it's something we just deal with. | 15:15 |
| duonghq | Jeffrey4l, ok | 15:15 |
| duonghq | Jeffrey4l, softlink in logging? | 15:16 |
| sdake | esharao indeed 17xx is incompatible with kubeadm | 15:16 |
| sdake | esharao the preflight checks fail when you use 17xx | 15:16 |
| sdake | rwellum see: https://review.openstack.org/451451 | 15:17 |
| sdake | rwellum i think for the guide we can just use kubernetes 1.5.5 | 15:17 |
| Jeffrey4l | duonghq, check this parameter http://docs.fluentd.org/v0.12/articles/out_file#symlinkpath | 15:17 |
| sdake | rwellum until the gate is sorted out | 15:17 |
| sdake | rwellum and iterate on the docs | 15:17 |
| *** Padouch has quit IRC | 15:17 | |
| esharao | ok.. starting my centos testing now | 15:17 |
| sdake | rwellum i really believe that review above will not require any kind of workarounds as kubeadm is actively tested with kubernetes 1.6.0 | 15:18 |
| *** rwallner has quit IRC | 15:18 | |
| duonghq | Jeffrey4l, thanks | 15:18 |
| rwellum | ok sdake will try | 15:20 |
| sbezverk | sdake: kubeadm is broken, at least version 2704 | 15:20 |
| sbezverk | rwellum: can you check your version of kubeadm? I need version lower than 2704 | 15:22 |
| openstackgerrit | Vikram Hosakote proposed openstack/kolla-kubernetes master: Deployment Guide Documentation https://review.openstack.org/447356 | 15:22 |
| *** pbandark`brb has quit IRC | 15:24 | |
| rwellum | https://www.irccloud.com/pastebin/9N36X8dO/ | 15:24 |
| sdake | sbezverk upstream admit to this - or your speculation? | 15:24 |
| rwellum | sbezverk: ^^^ | 15:24 |
| *** blallau has quit IRC | 15:24 | |
| kfox1111 | morning. | 15:24 |
| sdake | sup kfox1111 | 15:24 |
| sdake | kfox1111 well you got your wish - kubernretes 1.6.0 is in the rpm repos ;) | 15:24 |
| kfox1111 | sweet. | 15:24 |
| sbezverk | sdake: many people are experiemncing the same issue | 15:24 |
| kfox1111 | what broke? :) | 15:25 |
| manheim | if I am doing multinode deployment, is it absolutelly necesarry to do it with a registry server? | 15:25 |
| sbezverk | kfox1111: two things at least so far | 15:25 |
| sdake | kfox1111 kubeadm may be incompatible with 1.6.0 | 15:25 |
| manheim | can't i just pull from docker hub? | 15:25 |
| sdake | manheim you can pull from docker hub if yo ulike | 15:25 |
| kfox1111 | sdake: no 1.6.0 kubadm rpm yet? | 15:25 |
| sdake | manheim your performance will not be ideal | 15:25 |
| sdake | kfox1111 there is a 1.6.0 kubeadm rpm | 15:25 |
| manheim | seems fair :D | 15:25 |
| sdake | kfox1111 i've been using it for 3-4 days | 15:25 |
| sbezverk | kfox1111: there is and it breaks things | 15:25 |
| kfox1111 | sdake: there's been an alpha for a long time. | 15:25 |
| kfox1111 | sdake: should be a final release though. | 15:26 |
| kfox1111 | k | 15:26 |
| sdake | sbezverk if you want to validate your assumption you can use yum to install a specific version | 15:26 |
| sbezverk | sdake: it is not assuption, it a fact, I have the issue at the gate and on local cluster | 15:27 |
| sbezverk | kfox1111: can you run kubeadm version | 15:27 |
| sdake | i submitted a review for that sbezverk | 15:27 |
| sdake | here is this repo data: | 15:27 |
| sbezverk | version 2704 breaks init process | 15:27 |
| sdake | <version epoch="0" ver="1.6.0" rel="0"/> | 15:27 |
| sdake | there is a 1.6.0 without version 2704 | 15:28 |
| kfox1111 | sdake: ah. ok. | 15:28 |
| sdake | https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64/repodata/filelists.xml | 15:28 |
| kfox1111 | let me try and pull... | 15:28 |
| sdake | <package pkgid="5116fa4b73c700823cfc76b3cedff6622f2fbd0a3d2fa09bce6d93329771e291" name="kubeadm" arch="x86_64"> | 15:28 |
| sdake | <version epoch="0" ver="1.6.0" rel="0"/> | 15:28 |
| sdake | <file> | 15:28 |
| sdake | </file> | 15:28 |
| sdake | <file>/usr/bin/kubeadm</file> | 15:28 |
| sdake | </package> | 15:28 |
| sdake | <package pkgid="082436e6e6cad1852864438b8f98ee6fa3b86b597554720b631876db39b8ef04" name="kubeadm" arch="x86_64"> | 15:28 |
| sdake | <version epoch="0" ver="1.6.0" rel="0.alpha.0.2074.a092d8e0f95f52"/> | 15:28 |
| sdake | <file> | 15:28 |
| sdake | </file> | 15:28 |
| sdake | <file>/usr/bin/kubeadm</file> | 15:29 |
| sdake | </package> | 15:29 |
| sdake | there are two versions in the repository | 15:29 |
| sdake | 1.6.0-0 | 15:29 |
| sdake | and 1.6.0-0.alpha.0.2074.* | 15:29 |
| kfox1111 | -0? thats a bit odd. | 15:29 |
| sdake | agreeed its wierd | 15:29 |
| kfox1111 | maybe a testing rpm. | 15:29 |
| *** fooliouno has joined #openstack-kolla | 15:29 | |
| sdake | sup fooliouno | 15:29 |
| fooliouno | sdake: morning sdake | 15:29 |
| *** adrian_otto has quit IRC | 15:30 | |
| fooliouno | sdake: happy with the progress from yesterday. I see that you have newer versions of the guide | 15:30 |
| sdake | 1.6.0-0.alpha may take priority over 1.6.0-0 | 15:31 |
| sdake | with rpm | 15:31 |
| sdake | do we know the alpha is being installed int eh gate? | 15:31 |
| *** hrw has quit IRC | 15:31 | |
| sbezverk | kfox1111: there is also new parameter which needs to be added to kubelet, see this ps https://review.openstack.org/451391 | 15:32 |
| sdake | kfox1111 https://review.openstack.org/#/c/451451/1 | 15:32 |
| sdake | sbezverk kubeadm should be doing that work | 15:32 |
| sdake | or the kubelet packaging should be doing that work | 15:32 |
| sdake | sbezverk what you have there is a workaround for broken upstream packaging | 15:32 |
| sbezverk | sdake: at least I have a workaround, what do you suggest? | 15:33 |
| sdake | 2017-03-29 13:57:27.335915 | ---> Package kubeadm.x86_64 0:1.6.0-0.alpha.0.2074.a092d8e0f95f52 will be installed | 15:33 |
| sdake | sbezverk workaround is good | 15:33 |
| sdake | sbezverk helps fix the problem upstream in the upstream packaging if we can help them debug their mess | 15:34 |
| sdake | what I would suggest though is installing kubeadm-1.6.0-0 | 15:34 |
| sdake | let me give that a go | 15:34 |
| *** rwallner_ has quit IRC | 15:34 | |
| *** JoseMello has quit IRC | 15:35 | |
| openstackgerrit | Merged openstack/kolla master: make yum fail on missing packages https://review.openstack.org/449086 | 15:36 |
| *** hrw has joined #openstack-kolla | 15:36 | |
| sbezverk | sdake: it is the final release of kubeadm is broken | 15:36 |
| sdake | sbezverk on my local machinie: | 15:37 |
| sdake | [sdake@kolla tools]$ sudo rpm -qa | grep kubeadm | 15:37 |
| sdake | kubeadm-1.6.0-0.alpha.0.2074.a092d8e0f95f52.x86_64 | 15:37 |
| *** matrohon_ has quit IRC | 15:37 | |
| sdake | the gate has exactly the same version | 15:37 |
| sdake | note this version works for me locally | 15:37 |
| sdake | kbuernetes 1.6.0 upgrades probably break with kubeadm 1.6.0-0.alpha | 15:37 |
| sdake | i'd like to install just straight up kubeadm 1.6.0-0 | 15:38 |
| *** klindgren__ has joined #openstack-kolla | 15:38 | |
| *** klindgren_ has quit IRC | 15:40 | |
| *** sayantan_ has quit IRC | 15:41 | |
| *** sayantan_ has joined #openstack-kolla | 15:41 | |
| *** rwallner_ has joined #openstack-kolla | 15:42 | |
| inc0 | good morning everyone | 15:43 |
| spsurya | morning inc0 | 15:44 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Fixing 1.6.0 kubeadm init getting stuck https://review.openstack.org/451391 | 15:44 |
| sbezverk | inc0: good morning | 15:44 |
| *** sayantan_ has quit IRC | 15:45 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla stable/newton: make yum fail on missing packages https://review.openstack.org/451465 | 15:46 |
| duonghq | morning inc0 | 15:48 |
| sdake | kfox1111 sbezverk what i'd suggest is finding where that rpm package for kubeadm-1.6.0-0 is located on the internets | 15:48 |
| sdake | I know its out there | 15:48 |
| sdake | i gott aget my kid to school bbl | 15:48 |
| sbezverk | sdake: it turns out we need to specify version as 1.6.0-0 | 15:49 |
| sbezverk | then yum takes non aloha version | 15:49 |
| sbezverk | oops non alpha | 15:49 |
| sbezverk | version | 15:49 |
| sbezverk | I just pushed the fix to the gate to test | 15:50 |
| *** sayantan_ has joined #openstack-kolla | 15:50 | |
| *** rmart04 has quit IRC | 15:55 | |
| *** targon has quit IRC | 15:56 | |
| sdake | sbezverk pretty sure that is what I suggested :) | 15:56 |
| *** manheim has quit IRC | 15:57 | |
| sdake | the reason you need to specify the versio nto yum is because -0.alpha takes precedence over -0 | 15:57 |
| sdake | whoever did the versioning doesn't know how rpm works apprently :) | 15:57 |
| sbezverk | sdake: there might be an issue with kubernetes hosted mirrors as despite specifying complete version, gate still installs alpha one | 16:00 |
| *** vhosakot has joined #openstack-kolla | 16:00 | |
| kfox1111 | yeah. the alpha should be higher priority then -0 I think... | 16:00 |
| sdake | right - when I do yum install kubernetes-1.6.0-0 it thinks the alpha version is good enough | 16:01 |
| Jeffrey4l | meeting time, gusy. | 16:01 |
| sdake | the way to typically work aroudn this is with a yum install http://uri | 16:01 |
| sdake | I don't know what the uri is to the rpms | 16:01 |
| sdake | i looked in the repo location and don't immediately see the rpms | 16:02 |
| *** adrian_otto has joined #openstack-kolla | 16:03 | |
| sbezverk | man, what a PoS :( you resolve one issue to hit another.. | 16:03 |
| sbezverk | now kubeadm does not label nodes correctly, so canal's etcd cannot start | 16:04 |
| *** lamt has quit IRC | 16:10 | |
| *** rwallner_ has quit IRC | 16:12 | |
| *** dmsimard is now known as dmsimard|afk | 16:13 | |
| *** rwallner has joined #openstack-kolla | 16:13 | |
| kfox1111 | you can just psecify the version. | 16:16 |
| kfox1111 | yum install kubeadm-1.6.0-0 | 16:16 |
| *** rwallner has quit IRC | 16:17 | |
| kfox1111 | not seeing a kubeadm rpm yet... | 16:19 |
| kfox1111 | where did you find 1.6.0? | 16:19 |
| kfox1111 | oh... mayhbe its not printing it as its older. | 16:19 |
| kfox1111 | on an already installed system, yum downgrade kubeadm-1.6.0-0 | 16:21 |
| kfox1111 | kubeadm version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.0", GitCommit:"fff5156092b56e6bd60fff75aad4dc9de6b6ef37", GitTreeState:"clean", BuildDate:"2017-03-28T16:24:30Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} | 16:22 |
| kfox1111 | yup. thats newer. :/ | 16:22 |
| sbezverk | kfox1111: I used that exact command in the gate script | 16:22 |
| sbezverk | and got alpha still | 16:23 |
| sbezverk | kfox1111: I got it running on my cluster, by doing 2 changes | 16:23 |
| esharao | anyone using 1.6 | 16:23 |
| esharao | im running into issues when i do kubeadm -init | 16:24 |
| sbezverk | 1 - kubelet config chance | 16:24 |
| sbezverk | 2 - uninstalling alpha and installing normal kubeadm | 16:24 |
| sbezverk | esharao: please check this ps for the change you need to do BEFORE you run kubeadm init | 16:24 |
| sbezverk | https://review.openstack.org/451391 | 16:25 |
| kfox1111 | sbezverk: and the cgroup systemd thing? | 16:25 |
| kfox1111 | or is that really needed? I would have thought it would default to systemd | 16:25 |
| sbezverk | kfox1111: it is in the PS | 16:25 |
| sbezverk | one extra parameters for kubelet | 16:25 |
| kfox1111 | is it really needed though? | 16:25 |
| sbezverk | kfox1111: yep, otherwise kubelet fails to start | 16:26 |
| sbezverk | due to misconfiugration | 16:26 |
| sbezverk | kfox1111: see this https://github.com/kubernetes/kubernetes/issues/43805#issuecomment-290121607 | 16:27 |
| kfox1111 | sbezverk: maybe specify it as 1.6.0-0.x86_64 | 16:27 |
| kfox1111 | maybe its matching the .alpha still. | 16:27 |
| *** zhubingbing has quit IRC | 16:27 | |
| kfox1111 | sbezverk: ah. ok. thanks for the pointer. | 16:28 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Fixing 1.6.0 kubeadm init getting stuck https://review.openstack.org/451391 | 16:30 |
| kfox1111 | sbezverk: next ps, could you rename it to be something like upgrade to 1.6? we probably should land a few other things in the ps too, like updating canal to the 1.6 version. | 16:32 |
| sbezverk | kfox1111: you bet, new canal.yaml uses different labeling and node selection, not compatible with old one | 16:34 |
| esharao | sbezverk - following the fix 451391 .. it says unknown flag: --apiserver-advertise-address | 16:34 |
| kfox1111 | fun. | 16:34 |
| *** egonzalez has quit IRC | 16:34 | |
| kfox1111 | esharao: its failing cause it wasn't successfully downgrading kubeadm to the newer version. | 16:34 |
| kfox1111 | (parse that sentence carefully. :) | 16:34 |
| mnaser | Jeffrey4l i fired up the builds | 16:34 |
| sbezverk | esharao: it is because you are still using alpha vgersion | 16:34 |
| *** manheim has joined #openstack-kolla | 16:34 | |
| esharao | let me check.. | 16:35 |
| esharao | oh thats right.. | 16:35 |
| mnaser | building ubuntu-binary right now, i'll do {ubuntu,centos,oraclelinux}-{binary,source} | 16:35 |
| sbezverk | kfox1111: yeah back to the future ;) | 16:35 |
| kfox1111 | :) | 16:35 |
| mnaser | for 3.0.3 first and then ill do mitaka after | 16:35 |
| *** shardy has quit IRC | 16:35 | |
| Jeffrey4l | mnaser, thanks a lots | 16:35 |
| mnaser | np | 16:35 |
| *** duonghq has quit IRC | 16:36 | |
| sbezverk | esharao: http://paste.openstack.org/show/604702/ | 16:37 |
| Jeffrey4l | mnaser, what's your name in hub.docker.com? | 16:37 |
| esharao | got it.. working.. i just uninstalled and installed | 16:38 |
| Jeffrey4l | username or email. | 16:38 |
| mnaser | Jeffrey4l mnaser | 16:38 |
| hrw | https://etherpad.openstack.org/p/hrw-multiarch-and-arm64-containers will have more data about my work | 16:38 |
| mnaser | email is mnaser@vexxhost.com | 16:38 |
| Jeffrey4l | mnaser, added, you can push image to hub.docker.com/u/kolla now. | 16:38 |
| *** manheim has quit IRC | 16:38 | |
| mnaser | cool, i'll show a paste of all the images once they've all been built locally and once i have an ok that things seem ok | 16:39 |
| mnaser | ill push | 16:39 |
| * hrw off | 16:39 | |
| Jeffrey4l | great. | 16:40 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 uprgade https://review.openstack.org/451391 | 16:41 |
| *** shashank_t_ has quit IRC | 16:41 | |
| *** rwallner has joined #openstack-kolla | 16:42 | |
| *** srwilkers has quit IRC | 16:44 | |
| esharao | sbezverk: still see the same error - [apiclient] First node has registered, but is not ready yet | 16:45 |
| *** unicell has quit IRC | 16:46 | |
| sbezverk | esharao: kubeadm version | 16:46 |
| *** sean-k-mooney has quit IRC | 16:47 | |
| *** shashank_t_ has joined #openstack-kolla | 16:47 | |
| *** jistr|mtg is now known as jistr | 16:47 | |
| esharao | kubeadm version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.0", GitCommit:"fff5156092b56e6bd60fff75aad4dc9de6b6ef37", GitTreeState:"clean", BuildDate:"2017-03-28T16:24:30Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} | 16:49 |
| *** shardy has joined #openstack-kolla | 16:52 | |
| *** matrohon_ has joined #openstack-kolla | 16:53 | |
| *** rhallisey has quit IRC | 16:58 | |
| *** rhallisey has joined #openstack-kolla | 17:00 | |
| *** srwilkers has joined #openstack-kolla | 17:00 | |
| sdake | mnaser to build images | 17:01 |
| sdake | mnaser i typically run sudo pip install kolla==3.0.3 | 17:01 |
| esharao | anyone else successful with kubeadm 1.6? | 17:01 |
| sdake | mnaser then build them | 17:01 |
| sdake | mnaser (in other words, I dont git checkout to build images - I use the pip distributed version of kolla) | 17:02 |
| mnaser | sdake for "completeness" sake, i did pip install http://tarballs.o.o/kolla/kolla-3.0.3.tar.gz -- and building them now with different base and type | 17:02 |
| sdake | mnaser cool I dind't know that was a thing :) | 17:02 |
| esharao | brb | 17:02 |
| kfox1111 | esharao: focus is on getting it working in the gate right now I think. | 17:04 |
| *** bmaltais has joined #openstack-kolla | 17:04 | |
| openstackgerrit | Merged openstack/kolla stable/newton: Install uwsgi in Centos binary https://review.openstack.org/435687 | 17:04 |
| bmaltais | Good afternoon/morning | 17:05 |
| *** matrohon_ has quit IRC | 17:05 | |
| bmaltais | I successfully installed openstack using kolla following the great quick start guide and I have a question | 17:05 |
| bmaltais | I can't get the public network access to work | 17:06 |
| bmaltais | I assigned the same subnet as my primary eth0 interface and assigned a range of available public IP | 17:06 |
| bmaltais | but any attempt to snat from a VM of associate a floating IP does not appear to work | 17:07 |
| bmaltais | Is the private1 public network assocuate to the primary or the second NIC on the server? | 17:07 |
| bmaltais | I know just enough about openstack to be dangerous I am afraid | 17:08 |
| *** pbandark has joined #openstack-kolla | 17:08 | |
| sbezverk | kfox1111: do you know ny any chance where ubuntu store this file? 10-kubeadm.conf | 17:09 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 uprgade https://review.openstack.org/451391 | 17:09 |
| *** sayantan_ has quit IRC | 17:10 | |
| *** mrhillsman has quit IRC | 17:10 | |
| sbezverk | gents, who run kube on ubuntu please let me know the location of this file 10-kubeadm.conf | 17:10 |
| *** sambetts is now known as sambetts|afk | 17:11 | |
| *** mgoddard has quit IRC | 17:11 | |
| rwellum | sbezverk: https://github.com/kubernetes/release/rpm/10-kubeadm.conf ? | 17:12 |
| sbezverk | rwellum: on your local cluster | 17:12 |
| rwellum | Darn I think I killed that VM - 1s... | 17:13 |
| rwellum | Yeah sorry sbezverk | 17:14 |
| *** rwsu has joined #openstack-kolla | 17:15 | |
| kfox1111 | sbezverk: not off hand. | 17:15 |
| kfox1111 | only ubuntu k8s I've ever done is in the gate. | 17:15 |
| sbezverk | kfox1111: same here.. not sure where kubelet stores its startup file | 17:16 |
| sbezverk | on ubuntu | 17:17 |
| sbezverk | for now the goal is to get centos at least in the gate working again | 17:17 |
| kfox1111 | sbezverk: the gate script looks like it editied the same file on both distros. | 17:17 |
| sbezverk | kfox1111: nope it is multinode | 17:18 |
| kfox1111 | so they use to be the same at lest. | 17:18 |
| sbezverk | and multinode is centos only | 17:18 |
| *** vhosakot has quit IRC | 17:18 | |
| kfox1111 | sbezverk: thought you were talking about 10-kubeadm.conf | 17:18 |
| kfox1111 | there's no conditional in the gate based on distro, so should be the same for all? | 17:18 |
| sbezverk | kfox1111: there is condition right at the top | 17:18 |
| sbezverk | kfox1111: I meant in setup_kubernetes.sh script | 17:19 |
| kfox1111 | sbezverk: it ends at line 27 | 17:19 |
| openstackgerrit | Marcus Williams proposed openstack/kolla master: Add OpenDaylight Container to kolla https://review.openstack.org/416369 | 17:19 |
| sbezverk | kfox1111: it checkes if it is a master | 17:20 |
| kfox1111 | that ends on line 62 | 17:21 |
| sbezverk | please look at the latest revision of my ps | 17:21 |
| kfox1111 | k | 17:21 |
| *** signed8b_ is now known as signed8bit_Zzz | 17:21 | |
| kfox1111 | sbezverk: looks ok to me. | 17:23 |
| bmaltais | Any guide that show how to properly get a public network setup under kolla? | 17:23 |
| sbezverk | kfox1111: cool, let's see if this one will pass breaking point | 17:24 |
| bmaltais | The quickstart guide does not appear to implement a working public network | 17:24 |
| kfox1111 | should get farther. :) | 17:24 |
| kfox1111 | at least up until canal. :) | 17:25 |
| bmaltais | Tried modifying the init-runonce script but that did not work | 17:25 |
| *** athomas has quit IRC | 17:25 | |
| *** sayantan_ has joined #openstack-kolla | 17:26 | |
| *** unicell has joined #openstack-kolla | 17:27 | |
| *** sayantan_ has quit IRC | 17:27 | |
| *** sayantan_ has joined #openstack-kolla | 17:27 | |
| sbezverk | kfox1111: one step further but it stuck now of what other people reports, but I do not see it in my local cluster | 17:28 |
| sbezverk | kfox1111: [apiclient] First node has registered, but is not ready yet | 17:29 |
| *** MasterOfBugs has joined #openstack-kolla | 17:29 | |
| rwellum | bmaltais: not sure but i think that doc is out of date, refer to https://review.openstack.org/#/c/447356/19/doc/source/deployment-guide.rst - which is also a WIP. | 17:29 |
| bmaltais | rwellum: Thank's. I will read through it. | 17:32 |
| openstackgerrit | Marcus Williams proposed openstack/kolla-ansible master: Split Openvswitch into own role https://review.openstack.org/416793 | 17:32 |
| openstackgerrit | Tim Clark proposed openstack/kolla-kubernetes master: WIP - Update nova-control service charts for Ocata https://review.openstack.org/451495 | 17:34 |
| *** gfidente is now known as gfidente|afk | 17:36 | |
| *** tonanhngo has joined #openstack-kolla | 17:44 | |
| kfox1111 | sbezverk: looks like docker may need to be running before kubadm? | 17:45 |
| *** rhallisey has quit IRC | 17:46 | |
| *** tonanhngo_ has joined #openstack-kolla | 17:48 | |
| *** tonanhngo has quit IRC | 17:48 | |
| *** rwallner has quit IRC | 17:48 | |
| sbezverk | kfox1111: adding it.. | 17:51 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 uprgade https://review.openstack.org/451391 | 17:51 |
| *** pcaruana has quit IRC | 17:52 | |
| *** tonanhngo_ has quit IRC | 17:52 | |
| openstackgerrit | Dave Walker proposed openstack/kolla master: Allow the use of previously built parent images https://review.openstack.org/426322 | 17:53 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 uprgade https://review.openstack.org/451391 | 17:53 |
| *** tonanhngo has joined #openstack-kolla | 17:54 | |
| *** rhallisey has joined #openstack-kolla | 17:57 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: handle rtslib(-fb) package names and dependencies https://review.openstack.org/450805 | 18:00 |
| sbezverk | kfox1111: same thing on both ubuntu and centos, I added messages collection and syslog collection. hopefully when these jobs get killed, we can still access those logs to see WTF :( | 18:02 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 uprgade https://review.openstack.org/451391 | 18:04 |
| *** mgoddard has joined #openstack-kolla | 18:05 | |
| *** lamt has joined #openstack-kolla | 18:09 | |
| *** lamt has quit IRC | 18:10 | |
| *** rwallner has joined #openstack-kolla | 18:11 | |
| *** lamt has joined #openstack-kolla | 18:12 | |
| *** mgoddard has quit IRC | 18:13 | |
| *** bmaltais has quit IRC | 18:22 | |
| rwellum | sbezverk: kubeadm-1.6.0-0.x86_64 on centos did not get me past that cgroup issue | 18:23 |
| sbezverk | rwellum: even after you change kubelet start up config? | 18:24 |
| rwellum | sbezverk: must have missed a step then | 18:25 |
| *** pbandark has quit IRC | 18:25 | |
| *** fooliouno has quit IRC | 18:27 | |
| sbezverk | rwellum: I really hope so that you missed it otherwise we go back to square 1 | 18:27 |
| *** rwallner has quit IRC | 18:28 | |
| sbezverk | meeting time, bb in 2 hours | 18:28 |
| *** rwallner has joined #openstack-kolla | 18:30 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla-ansible master: Updated from global requirements https://review.openstack.org/451017 | 18:43 |
| *** openstackstatus has joined #openstack-kolla | 18:44 | |
| *** ChanServ sets mode: +v openstackstatus | 18:44 | |
| *** lucasxu has quit IRC | 18:45 | |
| *** lucasxu has joined #openstack-kolla | 18:45 | |
| rwellum | sbezverk: in case you see this during your meeting, with kubelet correctly run your work-aroung worked for me. | 18:48 |
| *** rwsu has quit IRC | 19:00 | |
| *** gfidente|afk has quit IRC | 19:03 | |
| *** common-- has joined #openstack-kolla | 19:07 | |
| esharao | rwellum did you get your kube cluster up? | 19:08 |
| rwellum | esharao: giving it a go now. | 19:09 |
| esharao | ok.. mine is not coming with 1.6 so let me know how it goes for you | 19:09 |
| rwellum | with this being the magic kubelet invocation: | 19:09 |
| rwellum | https://www.irccloud.com/pastebin/7UtR4uMb/ | 19:09 |
| *** rwallner has quit IRC | 19:11 | |
| common-- | Trying to follow https://docs.openstack.org/developer/kolla/newton/quickstart.html -- it does not appear to say to install kolla-ansible, so if you go through it, it will fail on kolla-genpwd | 19:12 |
| rwellum | esharao: which part do you fail on? | 19:13 |
| esharao | kubeadm init keeps on waiting.. - [apiclient] Created API client, waiting for the control plane to become ready | 19:14 |
| rwellum | sudo tail /var/log/messages ? | 19:16 |
| rwellum | https://www.irccloud.com/pastebin/MH1lrNN8/ | 19:16 |
| rwellum | Mine is stuck there too | 19:16 |
| esharao | kubelet cgroup driver: "systemd" is different from docker cgroup driver: "cgroupfs" | 19:16 |
| rwellum | Ah did you see the snippet above - with sbezverk fix for this? | 19:16 |
| rwellum | https://www.irccloud.com/pastebin/7UtR4uMb/ | 19:17 |
| *** signed8bit_Zzz is now known as signed8b_ | 19:17 | |
| esharao | one min | 19:17 |
| esharao | rechecked my config.. | 19:18 |
| esharao | the file has sbezverk's fix | 19:19 |
| rwellum | Hmmm I got past that, but stuck later on. | 19:19 |
| rwellum | with that fix. | 19:19 |
| *** david-lyle has quit IRC | 19:19 | |
| *** david-lyle has joined #openstack-kolla | 19:19 | |
| rwellum | You also used kubeadm-1.6.0-0.x86_64 ? | 19:19 |
| esharao | yes.. initially it went to alpha and then i uninstalled an specifically installed the 1.6.0 | 19:21 |
| rwellum | Docker version? I have: 1.12.6 | 19:22 |
| rwellum | Not sure if that's important tho | 19:22 |
| esharao | oh i have 17.03.1-ce | 19:24 |
| EmilienM | sdake, inc0: can you make sure we have some Kolla folks at the session from [deployment][forum] proposing a session about future of configuration management - ops + devs wanted! | 19:24 |
| rwellum | Would be the next step I think esharao , as the dep guide still says 1.12.6 - but I'm unsure because the 17.x is dockers new release model. Hopefully sdake or sbezverk can help here. | 19:26 |
| sdake | EmilienM inc0 said he would handle that forum request | 19:26 |
| sdake | EmilienM i think in terms of kolla folks - people go wher ethey want during summit - it was always challenging for me to get poeple to even come to htte kolla sessions that were kolla cores as ptl :) | 19:26 |
| esharao | hmm rwellum i believe eanylin got it up based on 17.x | 19:26 |
| esharao | but with kubeadm 1.5.5 | 19:26 |
| EmilienM | sdake: ack | 19:27 |
| *** david-lyle has quit IRC | 19:27 | |
| *** shashank_t_ has quit IRC | 19:28 | |
| *** david-lyle has joined #openstack-kolla | 19:30 | |
| openstackgerrit | Bertrand Lallau proposed openstack/kolla-ansible master: Trove fix backup restore with Swift https://review.openstack.org/451538 | 19:31 |
| *** matrohon_ has joined #openstack-kolla | 19:32 | |
| *** openstackgerrit has quit IRC | 19:33 | |
| *** openstackgerrit has joined #openstack-kolla | 19:33 | |
| openstackgerrit | Bertrand Lallau proposed openstack/kolla-ansible master: Trove fix backup restore with Swift https://review.openstack.org/451538 | 19:33 |
| rwellum | I'm willing to try that esharao but not sure how to get that version of docker yet. | 19:34 |
| esharao | https://docs.docker.com/engine/installation/linux/centos/ - and installed docker ce | 19:34 |
| esharao | if that is what you were asking | 19:35 |
| sdake | rwellum esharao re docker 17.xx - when I tried kubeadm 16.0.alpha - it failed the preflight checks with docker 1700 | 19:35 |
| sdake | so I had to install distro provided docker 12.6 | 19:36 |
| sdake | i have no idea what the behavior of kubeadm1 16.0 release is | 19:36 |
| sdake | as I have not tried it yet | 19:36 |
| esharao | sdake i used the https://review.openstack.org/#/c/451391 | 19:36 |
| sdake | rather 1.16.0-alpha and 1.16.0 release | 19:36 |
| esharao | and made change to 1.6.0-0 release by installing it with release name | 19:37 |
| sdake | i dont think it is wise to skip preflight checks esharao | 19:38 |
| esharao | kubeadm-1.6.0-0.x86_64 | 19:38 |
| sdake | preflight checks check compatibility and other various things | 19:38 |
| sdake | if they complain docker 17 is not supported, likely a reason for that | 19:38 |
| esharao | hmm sdake i tried it based on what sbezverk told | 19:38 |
| sdake | i couldn't get 13.0 to work at all even though kubeadm worked | 19:38 |
| esharao | hmm ok.. let me restart again with docker 1.12.6 and kubeadm 1.5.x and see how it goes | 19:39 |
| *** rwallner has joined #openstack-kolla | 19:39 | |
| openstackgerrit | Bertrand Lallau proposed openstack/kolla-ansible master: Set to Unix format trove-conductor.conf file https://review.openstack.org/451539 | 19:40 |
| *** lamt has quit IRC | 19:41 | |
| *** mcnanci has quit IRC | 19:45 | |
| sdake | kubeadm 1.6.0 may work with docker 1.12.6 or docker 17xx - just dont know | 19:53 |
| sdake | i used kubeadm 1.6.0-alpha to produce those docs | 19:53 |
| inc0 | sdake: there are few forum requests | 19:53 |
| *** matrohon_ has quit IRC | 19:54 | |
| inc0 | EmilienM: I'll be there for sure, thanks for heads up | 19:54 |
| EmilienM | cool | 19:54 |
| sdake | inc0 you said you would handle it - wfm - i was simply pointing out the call for forum requests was clsoing :) | 19:56 |
| *** Serlex has quit IRC | 19:56 | |
| rwellum | esharao: how do you get kubeadm 1.5.x? | 19:57 |
| sdake | guys - i would recommend kubeadm 1.6.0-release | 19:58 |
| esharao | did not.. that was yesterday.. :( | 19:58 |
| *** lamt has joined #openstack-kolla | 19:58 | |
| esharao | i used 1.6 and docker 1.12.6 | 19:58 |
| sdake | i would also recommend kubernetes 1.5.4 for now | 19:58 |
| esharao | but am running into an issue | 19:58 |
| esharao | [apiclient] First node has registered, but is not ready yet | 19:58 |
| sdake | until people sort out the issues with kubernetes 1.6 | 19:58 |
| rwellum | Yeah that's my issue esharao | 19:58 |
| esharao | :( | 19:59 |
| esharao | i see in my /var/log - Unable to update cni config | 20:00 |
| *** manheim has joined #openstack-kolla | 20:00 | |
| rwellum | Yeah that's it. | 20:01 |
| rwellum | I don't know the magic version combinations - I think I've tried 5 different combos.. | 20:01 |
| *** rwallner has quit IRC | 20:02 | |
| *** fooliouno has joined #openstack-kolla | 20:04 | |
| sbezverk | rwellum: thanks a lot Rich for confirming it.. | 20:06 |
| rwellum | sbezverk: but then I (and esharao ) hit another issue it seems. | 20:06 |
| rwellum | https://www.irccloud.com/pastebin/MH1lrNN8/ | 20:07 |
| *** pron has quit IRC | 20:07 | |
| srwilkers | yeah, that issues a pain in my ass | 20:07 |
| sbezverk | rwellum: same at the gate :( | 20:08 |
| esharao | ok.. so maybe i go back to kub 1.5.4 and try it again | 20:08 |
| *** pron has joined #openstack-kolla | 20:09 | |
| sbezverk | rwellum: by some reason kubelet is expecting to have networking up even before it got deployed.. | 20:09 |
| srwilkers | seems to be related to the changes made to the kubelet and how it's determining when a pod's ready | 20:09 |
| srwilkers | sbezverk, yep. nailed it | 20:09 |
| kfox1111 | back for a few. | 20:09 |
| rwellum | Yeah looks like it - nasty | 20:09 |
| srwilkers | with the changes, the master node never registers that its ready | 20:09 |
| *** rwallner has joined #openstack-kolla | 20:09 | |
| sbezverk | rwellum: there is an issue opened for the same reason with kube folks | 20:09 |
| sbezverk | srwilkers: the funny thing is, it worked one on my local test bed | 20:10 |
| sbezverk | I cannot see what I have done differently though | 20:10 |
| kfox1111 | the non cni plugin is still there. could it just be turned off for now? | 20:10 |
| kfox1111 | well, are we talking network plugin, or container engine plugin? | 20:11 |
| sbezverk | http://paste.openstack.org/show/604730/ | 20:11 |
| kfox1111 | sbezverk: nice. :) | 20:12 |
| sbezverk | kfox1111: nope, because I have no idea why it worked on my cluster | 20:12 |
| srwilkers | sbezverk, ive been fighting with it the better part of half a day with no luck | 20:13 |
| srwilkers | seems theres urgent interest on the issue in github to get it fixed, which is nice | 20:13 |
| sbezverk | srwilkers: yeah lots of folks reported the same issue | 20:13 |
| rwellum | A race condition maybe? Might explain why it works occasionally. | 20:14 |
| esharao | rwellum did you figure out how to install the previous version of kubeadm since 1.6 is broken | 20:20 |
| *** krtaylor has quit IRC | 20:20 | |
| sdake | esharao you can yum install the specific version | 20:21 |
| rwellum | No I was hoping you would. :) | 20:21 |
| esharao | oh i tried it.. | 20:21 |
| esharao | it did not work sdake | 20:21 |
| sdake | esharao didn't work as in didn't install or kubeadm didn't come up? | 20:21 |
| srwilkers | the problem is that the previous version of kubeadm still requires kubelet 1.6 | 20:21 |
| kfox1111 | sbezverk: bummer. | 20:21 |
| esharao | did not install (did not find) | 20:21 |
| srwilkers | which is whats causing the breaj | 20:21 |
| srwilkers | break, even | 20:21 |
| kfox1111 | sbezverk: ok, so.... does 1.5 still work? | 20:22 |
| kfox1111 | just wondering if the gate's broken. | 20:22 |
| kfox1111 | if 1.6 doesn't work for a few days, but the gate still works, not the end of the world. | 20:22 |
| kfox1111 | if the gate's broken, a whole nother thing. | 20:22 |
| sbezverk | kfox1111: we would need to pin 1.5.6 binaries then | 20:25 |
| sdake | if there is alot of interst upstream in kubernetes in fixing it | 20:26 |
| sbezverk | srwilkers: since you have the issue locally, could you try to remove KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin" | 20:26 |
| sdake | we shoud let them fix it - but provide what we know thus far | 20:26 |
| sbezverk | from kubelet command like | 20:26 |
| sbezverk | line.. | 20:26 |
| sdake | we can pin for a f ew days until a new release is cut or the docs are updated | 20:26 |
| rwellum | "previous version of kubeadm still requires kubelet 1.6" - that's an issue too right? | 20:27 |
| sdake | rwellum here are my deps: | 20:28 |
| *** Guest6666 has quit IRC | 20:28 | |
| *** Guest6666 has joined #openstack-kolla | 20:29 | |
| srwilkers | rwellum, yes. thats an issue | 20:29 |
| kfox1111 | sdake: yeah. it could easily take a few days to iron out by the current state of things. | 20:29 |
| kfox1111 | if thats the case, we should fix any breakage now by pinning back. | 20:29 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 uprgade https://review.openstack.org/451391 | 20:29 |
| kfox1111 | if we think we can fix 1.6 in the next couple hours though, we should probably just do that. | 20:30 |
| *** rwallner has quit IRC | 20:30 | |
| sbezverk | kfox1111: let's try to play with networking options | 20:30 |
| sbezverk | who has locally reproduced the issue and willing to try a few things? | 20:30 |
| rwellum | I can do it sbezverk - just a VM I can revert when it all goes belly-up. | 20:31 |
| kfox1111 | I'm fine pressing forward to 1.6 either way. just wondering when we shoudl cut our losses and just submit a gate fix for 1.5. | 20:31 |
| *** pramodrj07 has joined #openstack-kolla | 20:31 | |
| esharao | what do you want to try sbezverk | 20:32 |
| sbezverk | kfox1111: let's try to workaround it until the eob | 20:32 |
| kfox1111 | sbezverk: kk. sounds good. | 20:32 |
| sdake | rwellum having trouble logging into my local network - when i can I'll list out my deps in a few minutes | 20:32 |
| *** david-lyle has quit IRC | 20:33 | |
| sdake | sbezverk so your planning to get gate working with kubernetes 1.6 then, or pin to 1.5.5? | 20:33 |
| sbezverk | esharao: rwellum: find kubelet startup file and make sure this env variable KUBELET_NETWORK_ARGS is romoved | 20:33 |
| rwellum | sdake: no worries, but we are in addition having issues installing older versions - if you have the secret sauce for that it would help. | 20:33 |
| sbezverk | sdake: yes | 20:33 |
| sdake | rwellum indeed I do have the secret sauce | 20:33 |
| *** MasterOfBugs has quit IRC | 20:34 | |
| sdake | rwellum its not secret - just have to know how to use rpm :) | 20:34 |
| sdake | rwellum keep in mind by use - i mean be expert in - i worked at rht for 9 long long years | 20:34 |
| sdake | thats a 64bit there ;) | 20:34 |
| rwellum | cool sdake | 20:35 |
| *** rwallner has joined #openstack-kolla | 20:35 | |
| *** david-lyle has joined #openstack-kolla | 20:35 | |
| rwellum | most people I know went the other direction... | 20:35 |
| rwellum | voluntarily and not so. | 20:35 |
| sdake | rwellum which other direction | 20:35 |
| sdake | you mean deb? | 20:36 |
| rwellum | from your place of work to rh | 20:36 |
| sdake | not surprising - people change employers in our indsutry from one to another | 20:36 |
| sdake | and even back again :) | 20:36 |
| sdake | rht is a great place to work - i enjoyed it | 20:37 |
| sdake | cisco is also a great place to work - i enjoy it | 20:37 |
| sdake | nothing really bad to say about either co | 20:37 |
| rwellum | lucky either way | 20:38 |
| sbezverk | gents, are you trying to reconfigure kubelet or enjoying sdake speach ;) | 20:38 |
| rwellum | sbezverk: I am trying to find KUBELET_NETWORK_ARGS | 20:38 |
| srwilkers | ;) | 20:39 |
| sdake | I'll tell you i don't regret getting off the linux desktop :) | 20:39 |
| sdake | http://paste.openstack.org/show/604736 | 20:39 |
| srwilkers | thats not secret sauce, thats running the old version of kubelet that isnt broken | 20:40 |
| *** david-lyle has quit IRC | 20:40 | |
| sdake | srwilkers yes - I didn't say it was secret sauce | 20:40 |
| esharao | sbezverk it worked | 20:41 |
| esharao | but kubectl get pods is not connecting | 20:41 |
| *** david-lyle has joined #openstack-kolla | 20:42 | |
| sbezverk | esharao: one sec what worked? | 20:42 |
| esharao | my kubeadm init worked | 20:42 |
| sbezverk | when you do kubectl get pods what do you get? | 20:43 |
| esharao | http://paste.openstack.org/show/604737/ | 20:43 |
| esharao | thats the problem - kubectl says cannot connect to 8080 | 20:43 |
| sbezverk | esharao: that is notmal | 20:43 |
| esharao | The connection to the server localhost:8080 was refused - did you specify the right host or port? | 20:43 |
| sbezverk | you need to change manifest | 20:43 |
| esharao | oh is it inyour patch? | 20:44 |
| sbezverk | vi /etc/kubernetes/manifest/kube-apiserver.yaml | 20:44 |
| *** cu5 has joined #openstack-kolla | 20:44 | |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Pin gate to working versions of kubernetes 1.5.4 and kubeadm https://review.openstack.org/451556 | 20:44 |
| sdake | rwallner that command should work - might wait for the gate to do its thing | 20:45 |
| sdake | rwallner i really dont want to uninstall/reinstall to test it as i have working functional copies ;) | 20:45 |
| sbezverk | add this line2 | 20:45 |
| sbezverk | - --insecure-port=8080 | 20:45 |
| sbezverk | - --insecure-bind-address=0.0.0.0 | 20:45 |
| sdake | rwellum although the gate should return successful results for centos if that works ^^ | 20:45 |
| sbezverk | after line - --kubelet-client-key= | 20:46 |
| *** david-lyle has quit IRC | 20:46 | |
| esharao | done | 20:47 |
| *** rwallner has quit IRC | 20:47 | |
| sbezverk | esharao: when you exit vi, apiserver should automatically gets restarted | 20:48 |
| sbezverk | just give it a couple of minutes | 20:48 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Pin gate to working versions of kubernetes 1.5.4 and kubeadm https://review.openstack.org/451556 | 20:48 |
| sbezverk | and watch for port 8080 listening on 0.0.0.0 | 20:48 |
| esharao | http://paste.openstack.org/show/604738/ | 20:49 |
| esharao | its working now | 20:49 |
| sbezverk | :-) | 20:49 |
| sbezverk | now we need to see if we can get networking running again | 20:49 |
| esharao | :) | 20:50 |
| *** rwsu has joined #openstack-kolla | 20:50 | |
| esharao | want me to try the canal stuff? | 20:50 |
| sbezverk | esharao: you need to wget this file https://raw.githubusercontent.com/projectcalico/canal/master/k8s-install/kubeadm/1.6/canal.yaml | 20:51 |
| sbezverk | esharao: then change whatever ip info you want in there aand then we will try to create canal thingy.. | 20:51 |
| esharao | yep doing that | 20:52 |
| *** david-lyle has joined #openstack-kolla | 20:52 | |
| *** rwallner has joined #openstack-kolla | 20:53 | |
| rwellum | sdake: do we need to add that patch to the dep doc? | 20:53 |
| *** srwilkers has quit IRC | 20:53 | |
| esharao | sbezverk - all running | 20:54 |
| sbezverk | you mean all or all all all ;)? | 20:54 |
| esharao | http://paste.openstack.org/show/604740/ | 20:55 |
| sbezverk | esharao: that is a beautyyyyyyyy | 20:55 |
| esharao | :D | 20:55 |
| *** cu5 has quit IRC | 20:55 | |
| esharao | so continue with kolla? | 20:56 |
| esharao | following sdake DG | 20:56 |
| *** rwallner has quit IRC | 20:57 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 uprgade https://review.openstack.org/451391 | 21:01 |
| sbezverk | let's see if this recepie helps in the gate | 21:01 |
| esharao | sbezverk - any idea on taint nodes failing? | 21:02 |
| sbezverk | esharao: you need to check kube 1.6.0 change log, I think I saw something there about taints, do not remember what exactly | 21:03 |
| esharao | ok.. | 21:04 |
| *** cu5 has joined #openstack-kolla | 21:04 | |
| hawi | question: is there any chance to debug why mariadb doesnt come up properly after "Waiting for MariaDB service to be ready"? in somehow one moment all mariadb instances were not listening on 3306 port at all. i have no idea how it happened, but i really dont want to reinstall it, i want to understand, what's wrong with it | 21:06 |
| *** rwsu has quit IRC | 21:06 | |
| *** mgoddard has joined #openstack-kolla | 21:06 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 uprgade https://review.openstack.org/451391 | 21:07 |
| *** rhallisey has quit IRC | 21:08 | |
| SamYaple | hawi: if the cluster died then it can't recover on its own under most situations | 21:09 |
| *** jrobinson has joined #openstack-kolla | 21:09 | |
| SamYaple | hawi: it could have been network disruption on your end that caused the nodes to lose quorum, but if no node is listening the clsuter is down right now | 21:09 |
| SamYaple | youll have to run the recovery playbook | 21:09 |
| hawi | SamYaple: i tried, nothing happened | 21:09 |
| sbezverk | esharao: kubectl taint nodes kolla.sbezverk.cisco.com node-role.kubernetes.io/master:NoSchedule- | 21:09 |
| sbezverk | it is new in 1.6.0 | 21:10 |
| SamYaple | hawi: the recovery playbook expects all running mariadb docker containers to be stopped | 21:10 |
| esharao | ahh ok.. | 21:10 |
| hawi | SamYaple: you mean mariadb_recovery yes? | 21:10 |
| esharao | no schedule.. | 21:10 |
| SamYaple | hawi: yes | 21:10 |
| rwellum | sdake: pinning like you suggested worked for me | 21:10 |
| rwellum | http://paste.openstack.org/show/604742/ | 21:10 |
| esharao | that did the trick sbezverk | 21:11 |
| hawi | SamYaple: i'm really impressed! in somehow it didnt work for several hours back | 21:13 |
| *** mgoddard has quit IRC | 21:14 | |
| hawi | but now ... it's good... | 21:14 |
| SamYaple | hawi: i hacked your system and fixed it while you werent looking | 21:14 |
| *** krtaylor has joined #openstack-kolla | 21:14 | |
| hawi | :) | 21:14 |
| hawi | dunno...perhaps i didnt stop containers. i got compains about my.cnf and galera.j2 differ, also about gcs connect failed: Connection timed out | 21:16 |
| SamYaple | stopping the containers is a must if its broken like you described | 21:17 |
| *** rwsu has joined #openstack-kolla | 21:20 | |
| *** jtriley has quit IRC | 21:24 | |
| *** yee379 has joined #openstack-kolla | 21:25 | |
| *** lp|Kubecon has quit IRC | 21:25 | |
| *** kolla-slack has quit IRC | 21:25 | |
| *** kolla-slack1 has joined #openstack-kolla | 21:25 | |
| *** yee37915 has quit IRC | 21:26 | |
| *** eanylin has quit IRC | 21:26 | |
| *** eanylin has joined #openstack-kolla | 21:26 | |
| *** mattmceuen has quit IRC | 21:28 | |
| *** lazyPower has joined #openstack-kolla | 21:28 | |
| *** lucasxu has quit IRC | 21:34 | |
| hawi | SamYaple: thanks. didnt expect so simple solution, as i already spend couple of hours on trying to figure out what's wrong | 21:34 |
| fooliouno | I tried to create a VM, but got an error: "No valid host was found. There are not enough hosts available" | 21:36 |
| fooliouno | However, there is a node listed under hypervisors | 21:37 |
| eanylin | I just got my Cirros VM up :) | 21:37 |
| fooliouno | enaylin: any changes needed? | 21:37 |
| fooliouno | eanylin: congrats :) | 21:37 |
| eanylin | my kubeadm version is v1.6.0-alpha.0.2074+a092d8e0f95f52 | 21:37 |
| inc0 | fooliouno: check nova scheduler log | 21:37 |
| eanylin | docker version is v1.6.0-alpha.0.2074+a092d8e0f95f52 | 21:37 |
| eanylin | 17.03.0-ce | 21:38 |
| eanylin | sorry | 21:38 |
| fooliouno | inc0: Let me check that | 21:38 |
| eanylin | docker version is 17.03.0-ce | 21:38 |
| sdake | eanylin grats ddue :) | 21:38 |
| sdake | eanylin replicated the gate outside the gate ;) | 21:38 |
| sdake | eanylin mind adding any comments to the review - and i'll sort em out tonight? | 21:39 |
| eanylin | sdake: I had to do what is mentioned here https://bugs.launchpad.net/packstack/+bug/1661267 | 21:39 |
| openstack | Launchpad bug 1661267 in Packstack "libvirtd crashes while launching an instance from openstack horizon" [High,Fix released] | 21:39 |
| sdake | so we can get this documentation merged | 21:39 |
| eanylin | I believe we may still hit problems due to https://bugs.launchpad.net/packstack/+bug/1661267 | 21:39 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 uprgade https://review.openstack.org/451391 | 21:39 |
| eanylin | sdake: sure | 21:40 |
| sdake | cpu_mode=none | 21:40 |
| * sdake GROANS | 21:40 | |
| eanylin | yes | 21:40 |
| eanylin | it broke my hypervisor connection | 21:40 |
| eanylin | because of that | 21:40 |
| eanylin | I dont know if it will work properly if its on bare metal, didnt have one to test | 21:41 |
| eanylin | I am using Centos7 VM and had this problem | 21:41 |
| *** pramodrj07 has quit IRC | 21:41 | |
| *** pramodrj07 has joined #openstack-kolla | 21:41 | |
| eanylin | had to manually update nova.conf in nova-compute to get around that | 21:41 |
| *** mcnanci has joined #openstack-kolla | 21:42 | |
| *** schwicht has quit IRC | 21:43 | |
| *** ruhe has quit IRC | 21:44 | |
| *** timss has quit IRC | 21:44 | |
| *** timss has joined #openstack-kolla | 21:45 | |
| fooliouno | inc0: nova-scheduler says that the filter RetryFilter returned 0 hosts | 21:48 |
| inc0 | fooliouno: hmm...from top of my head I can't remember reason for that | 21:48 |
| inc0 | check nova-compute log | 21:48 |
| fooliouno | ok .. | 21:48 |
| inc0 | I have one candidate in mind | 21:49 |
| *** ruhe has joined #openstack-kolla | 21:49 | |
| eanylin | fooliouno: Did you set up the cell0 already? | 21:50 |
| *** shardy has quit IRC | 21:50 | |
| fooliouno | enaylin: the cell0-create-db pod completed | 21:51 |
| eanylin | how about nova-api-create-simple-cell-job ? | 21:53 |
| *** erlon has quit IRC | 21:55 | |
| *** sayantan_ has quit IRC | 21:55 | |
| fooliouno | eanylin: I see a nova-api-create-cell (not simple-cell) and that job completed successfully | 21:55 |
| eanylin | ok | 21:56 |
| fooliouno | eanylin: I need to head home, but will login later in the evening. | 21:56 |
| eanylin | fooliouno: Sure | 21:57 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 uprgade https://review.openstack.org/451391 | 21:58 |
| *** fooliouno has quit IRC | 21:59 | |
| *** esharao has quit IRC | 22:00 | |
| *** sayantan_ has joined #openstack-kolla | 22:04 | |
| *** schwicht has joined #openstack-kolla | 22:09 | |
| *** adrian_otto has quit IRC | 22:09 | |
| *** adrian_otto has joined #openstack-kolla | 22:10 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 uprgade https://review.openstack.org/451391 | 22:18 |
| *** sayantan_ has quit IRC | 22:21 | |
| *** sayantan_ has joined #openstack-kolla | 22:21 | |
| *** rwsu has quit IRC | 22:23 | |
| *** manheim has quit IRC | 22:32 | |
| *** shashank_t_ has joined #openstack-kolla | 22:33 | |
| *** rwsu has joined #openstack-kolla | 22:35 | |
| *** sayantan_ has quit IRC | 22:36 | |
| *** krtaylor has quit IRC | 22:38 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 uprgade https://review.openstack.org/451391 | 22:38 |
| sbezverk | kfox1111: almost there :) last piece, fixing tainting.. | 22:41 |
| *** schwicht has quit IRC | 22:47 | |
| *** shashank_t_ has quit IRC | 22:54 | |
| *** shashank_t_ has joined #openstack-kolla | 22:55 | |
| *** sayantan_ has joined #openstack-kolla | 22:58 | |
| kfox1111 | back. | 22:58 |
| kfox1111 | sbezverk: nice. :) | 22:58 |
| *** shashank_t_ has quit IRC | 22:59 | |
| sbezverk | kfox1111: they also changed token format, man it is crazy.. | 23:00 |
| kfox1111 | sbezverk: https://raw.githubusercontent.com/projectcalico/canal/master/k8s-install/kubeadm/1.6/canal.yaml ? | 23:00 |
| kfox1111 | heh. yeah. I heard that. | 23:00 |
| kfox1111 | one of the drawbacks on using something thats alpha. | 23:00 |
| kfox1111 | (now beta) | 23:00 |
| sbezverk | kfox1111: http://logs.openstack.org/91/451391/17/check/gate-kolla-kubernetes-deploy-centos-binary-3-ceph-multi-nv/5ddb440/console.html#_2017-03-29_22_48_46_109945 | 23:00 |
| kfox1111 | sbezverk: did we fail to parse out the token? | 23:02 |
| sbezverk | yep | 23:02 |
| sbezverk | but they give reg exp | 23:02 |
| sbezverk | "^([a-z0-9]{6})\\.([a-z0-9]{16})$" | 23:02 |
| sbezverk | I will use it instead | 23:02 |
| sbezverk | of what we have now | 23:02 |
| kfox1111 | hmm.... | 23:02 |
| kfox1111 | no, I think that will break... | 23:02 |
| kfox1111 | give me a sec... | 23:02 |
| sbezverk | ok | 23:03 |
| kfox1111 | so, we have: kubeadm join --token 6aba5e.876011764b2ae7b7 10.32.180.247:6443 | 23:03 |
| kfox1111 | oh... | 23:04 |
| kfox1111 | they removed '=' from the string. | 23:04 |
| kfox1111 | used to be --token=.... | 23:04 |
| kfox1111 | now its --token .... | 23:04 |
| kfox1111 | that makes it easier to parse. :) | 23:04 |
| kfox1111 | so, on line 62, drop out: | sed 's/[^=]*=//' | 23:05 |
| kfox1111 | change $3 to $4, | 23:05 |
| kfox1111 | and $4 to $5. | 23:05 |
| kfox1111 | should fix it. | 23:06 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 uprgade https://review.openstack.org/451391 | 23:07 |
| sbezverk | kfox1111: ok let me try it | 23:08 |
| sbezverk | kfox1111: but the format is blah.blahblah now | 23:09 |
| sbezverk | ah ok it will grab whole token | 23:10 |
| kfox1111 | right. :) | 23:11 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 uprgade https://review.openstack.org/451391 | 23:13 |
| *** pramodrj07 has quit IRC | 23:22 | |
| kfox1111 | 2017-03-29 23:22:55.348903 | Error from server (Forbidden): User "system:node:centos-7-osic-cloud1-s3700-8170757" cannot list jobs.batch in the namespace "kube-system". (get jobs.batch) | 23:25 |
| kfox1111 | fun... | 23:25 |
| kfox1111 | sbezverk: might want to disable rbac for now. | 23:26 |
| sbezverk | kfox1111: man, will it eve stop ?!?! | 23:27 |
| kfox1111 | hehe. a lot of new things in 1.6. | 23:27 |
| kfox1111 | but so far, less then a day to fix. not too bad really. | 23:27 |
| sbezverk | kfox1111: well we do not know what will hit us next | 23:29 |
| kfox1111 | thats why we were talking about putting a pin back to 1.5 just in case this is too deep a rabbit hole. | 23:30 |
| kfox1111 | https://github.com/tailnode/kubernetes.github.io/commit/1c04cedd412b31bbcd8b708e8ac1033fac8e31a6 | 23:30 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 uprgade https://review.openstack.org/451391 | 23:31 |
| kfox1111 | looks like you can set authorization mode but not sure you can set it via the commandline. | 23:31 |
| kfox1111 | will that work? | 23:31 |
| sbezverk | kfox1111: yeah if by the end of day is not working, then go back to 1.5 | 23:32 |
| *** jtriley has joined #openstack-kolla | 23:32 | |
| kfox1111 | does rbac need to be replaced with the allow plugin? or something different? | 23:32 |
| kfox1111 | do you have a copy of kube-apiserver.yaml somewhere? | 23:33 |
| sbezverk | kfox1111: I removed rbac and I think it should fall back | 23:33 |
| sbezverk | kfox1111: sure | 23:33 |
| kfox1111 | it may need to be --authorization-mode=AlwaysAllow | 23:33 |
| sbezverk | http://paste.openstack.org/show/604756/ | 23:35 |
| kfox1111 | cool. thx. | 23:35 |
| kfox1111 | yeah. I'm guessing the sed should be /RBAC/AlwaysAllow/ but I guess we'll see. :) | 23:36 |
| kfox1111 | I'm a little surprised its using system:node:centos-7-osic-cloud1-s3700-8170757 though... | 23:37 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 uprgade https://review.openstack.org/451391 | 23:38 |
| kfox1111 | I'd expect it to be the admin by default... | 23:38 |
| kfox1111 | kind of weird. | 23:38 |
| sbezverk | kfox1111: strange on my cluster I can run command even with rbac | 23:39 |
| kfox1111 | yeah, kind of weirded helm would be tryign to use the node creds. | 23:39 |
| *** lamt has quit IRC | 23:39 | |
| kfox1111 | I'm guessing we will hit a bunch of other problems with rbac enabled though. | 23:39 |
| kfox1111 | so turning it off for now is agood thing. | 23:40 |
| sbezverk | but I changed to AlwaysAllow | 23:40 |
| kfox1111 | should have a whole different ps for figuring out how to enable it. | 23:40 |
| kfox1111 | maybe a set of ps's. | 23:40 |
| *** schwicht has joined #openstack-kolla | 23:40 | |
| sdake | i've got a patch in the queue for 1.5 | 23:46 |
| sdake | don't have ubuntu tho - so i guess i'll sort that out in some way | 23:47 |
| *** hieulq has quit IRC | 23:47 | |
| sdake | https://review.openstack.org/#/c/451556/ | 23:47 |
| sbezverk | kfox1111: it went further :) | 23:49 |
| kfox1111 | you sure? one of them failed. | 23:50 |
| kfox1111 | centos-binary-2-ceph-nv failed.... maybe random... | 23:50 |
| kfox1111 | which one you looking at? | 23:51 |
| *** adrian_otto has quit IRC | 23:51 | |
| kfox1111 | ovs failed too... | 23:51 |
| kfox1111 | hmm.. thought that one did get farther... | 23:51 |
| sbezverk | kfox1111: we should use port :8080 for inside communication | 23:52 |
| kfox1111 | not sure I follow. | 23:53 |
| kfox1111 | http://logs.openstack.org/91/451391/21/check/gate-kolla-kubernetes-deploy-centos-binary-2-external-ovs-nv/b16b427/logs/pods.yaml is very empty... | 23:53 |
| kfox1111 | weird. | 23:53 |
| sbezverk | kfox1111: because it tries to connect to 6443 | 23:54 |
| kfox1111 | which thingy? | 23:59 |
| kfox1111 | kubectl should use whatever kubeadm configured by default? | 23:59 |
| kfox1111 | it always has in the past at least. | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!