Wednesday, 2017-03-29

*** shashank_t_ has quit IRC00:01
sdakeyou always know when an engine blows a man can't win em all00:02
sdakeon that note - dinner time after torture session00:02
*** deadnull_ has quit IRC00:03
*** yangyapeng has joined #openstack-kolla00:03
*** yangyape_ has quit IRC00:03
*** Pavo has joined #openstack-kolla00:05
*** goldyfruit has quit IRC00:07
*** schwicht has joined #openstack-kolla00:19
*** yangyape_ has joined #openstack-kolla00:20
*** yangyapeng has quit IRC00:20
*** dave-mccowan has joined #openstack-kolla00:22
*** Pavo has quit IRC00:24
*** yingjun has joined #openstack-kolla00:25
*** Pavo has joined #openstack-kolla00:26
*** fooliouno has joined #openstack-kolla00:32
openstackgerritZeyu Zhu proposed openstack/kolla-kubernetes master: Remove useless modules  https://review.openstack.org/45066200:38
*** duonghq_ is now known as duonghq00:38
duonghqmorning guys00:39
duonghqsdake, nice guide, I'm trying again with AIO first00:40
duonghqhope that it's ok00:40
*** zhuzeyu has joined #openstack-kolla00:47
*** tovin07_ has joined #openstack-kolla00:59
*** Pavo has quit IRC01:02
*** MarginHu has joined #openstack-kolla01:04
*** yangyape_ has quit IRC01:06
*** cuongnv has joined #openstack-kolla01:09
*** eanylin has quit IRC01:11
*** adrian_otto has quit IRC01:18
*** zhubingbing_ has joined #openstack-kolla01:20
fooliounosdake: All my pods came up on a 2 node setup now. nova-compute is up and running as well. Thanks to you and everyone else who put the guide together.01:20
*** david-lyle has joined #openstack-kolla01:24
fooliounoDoes anyone know where and how to specify a public endpoint for Horizon. It did not come up with one.01:26
*** furlongm has quit IRC01:28
openstackgerritjimmygc proposed openstack/kolla-ansible master: Split globals.yml into mutiple part by service  https://review.openstack.org/45010601:32
*** celo_r has joined #openstack-kolla01:33
openstackgerritjimmygc proposed openstack/kolla-ansible master: Split globals.yml into mutiple part by service  https://review.openstack.org/45010601:39
*** caowei has joined #openstack-kolla01:41
*** shashank_t_ has joined #openstack-kolla01:44
*** eanylin has joined #openstack-kolla01:50
openstackgerritMerged openstack/kolla-ansible master: Heat: add oslo_messaging_notifications config  https://review.openstack.org/44451501:56
*** pramodrj07 has quit IRC01:58
*** MasterOfBugs has quit IRC01:58
*** dixiaoli has joined #openstack-kolla01:58
*** zhurong has joined #openstack-kolla02:04
*** yangyapeng has joined #openstack-kolla02:05
*** zhubingbing_ has quit IRC02:07
*** zhubingbing has joined #openstack-kolla02:10
*** unicell has quit IRC02:12
openstackgerritZeyu Zhu proposed openstack/kolla-kubernetes master: Fix error in running-tests.rst  https://review.openstack.org/45114102:12
sdakefooliouno the guide works for you then?02:13
sdakefooliouno are you able to start a nova vm?02:14
sdakefooliouno we htink something is missing related to starting various helm charts and it hsoudl be pretty easy to find02:14
*** fooliouno_ has joined #openstack-kolla02:16
fooliouno_sdake: openstack cli commands are being authenticated against the VIP address and they are failing. Troubleshooting them right now.02:16
*** celo_r has quit IRC02:17
*** celo_r has joined #openstack-kolla02:17
*** fooliouno has quit IRC02:17
*** celo_r has quit IRC02:18
*** adrian_otto has joined #openstack-kolla02:22
*** goldyfruit has joined #openstack-kolla02:24
*** yangyapeng has quit IRC02:25
fooliouno_It appears that the keystone-admin address is being set to the kolla_internal_vip_address in globals.yml. Does that seem right?02:27
*** adrian_otto has quit IRC02:27
*** shashank_t_ has quit IRC02:33
*** yangyapeng has joined #openstack-kolla02:33
*** david-lyle has quit IRC02:36
openstackgerritOpenStack Proposal Bot proposed openstack/kolla-ansible master: Updated from global requirements  https://review.openstack.org/45101702:40
fooliouno_eanylin: Do you still set the kolla_internal_vip_address in globals.yml?02:47
*** yuanying has quit IRC02:55
duonghqfooliouno_, can you provide some log, it seems that same here02:55
eanylinfooliouno: I did set the internal_vip_address in globals.yml02:56
eanylinsame IP as what I set for Kolla Ansible02:57
fooliouno_eanylin: Thanks! I had it set, but had errors, so was just wondering.02:59
fooliouno_duonghq: https://pastebin.com/FZiDP7G602:59
eanylinfooliouno: My initial set up has keystone internal same as keystone public02:59
eanylinI am trying your settings02:59
eanylinwill see what happens02:59
fooliouno_duonghq: The 10.124.11.200 is an unused IP that is in the same subnet as my 2 nodes. 10.124.11.29/2303:01
*** dave-mccowan has quit IRC03:03
fooliouno_I set the 10.124.11.200 as my kolla_internal_vip_address03:05
duonghqfooliouno_, did you try set VIP to one of kubeapi server?03:10
duonghqdue to we are diabling haproxy03:10
zhubingbinghi guys03:11
zhubingbing;)03:11
duonghqsup zhubingbing03:12
openstackgerritzhongshengping proposed openstack/kolla-ansible master: Remove verbose option  https://review.openstack.org/45115503:13
zhubingbingnice meet you duonghq03:13
fooliouno_duonghq: The IP of the node that the kube-apiserver is running on?03:13
zhubingbingyour kolla-kubernetes cluster is be ok?03:13
duonghqzhubingbing, not yet03:14
duonghqlast night I bring it to better state03:14
zhubingbing;)03:14
zhubingbingi try to do it agin03:14
duonghqfooliouno_, yesss, did you try AIO deployment03:14
zhubingbingtoday03:14
duonghqzhubingbing, nice03:15
fooliouno_I have a 2 node deployment with a kube master and the second node where all the kolla containers are deployed03:15
duonghqfooliouno_, ok03:15
fooliouno_duonghq: I will try using that IP when I bring them up again.03:16
*** yangyapeng has quit IRC03:17
duonghqfooliouno_, nice03:18
*** dixiaoli has quit IRC03:18
eanylinfooliouno: I correct my earlier statement, that internal_vip was my kube-proxy container IP03:20
eanylinthe one for kolla ansible has to be an IP that is not being used03:21
eanylinso they are not the same03:21
*** dixiaoli has joined #openstack-kolla03:26
duonghqzhubingbing, ping03:28
zhubingbinghear03:29
openstackgerritzhongshengping proposed openstack/kolla-ansible master: Remove min_l3_agents_per_router option  https://review.openstack.org/45115903:29
*** goldyfruit has quit IRC03:34
*** erlon has quit IRC03:35
*** david-lyle has joined #openstack-kolla03:37
fooliouno_eanylin: thanks for checking. Something in my configuration seems to be off. openstack cli commands dont work as they are directed to the unused IP.03:39
*** zhubingbing has quit IRC03:42
eanylinfooliouno: Ok. No problem.03:43
eanylinprobably remove the config under /etc/kolla and re-generate?03:44
fooliouno_eanylin: I tried that as well as cleaning up everything and bringing it up again. Still no luck.03:46
eanylinhmm03:47
eanylinwhat if you use keystone internal same as keystone public?03:47
*** jrobinson has quit IRC03:52
fooliouno_I can try setting that to see if it works. Will give it a shot tomorrow.03:53
eanylinfooliouno: Ok03:53
eanylinWasnt sure if you did the part on placement service/keystone endpoints creation, I think I figured out why it was crashing for me earlier03:55
duonghqanybody got "Failed to contact the endpoint at http://keystone-internal:5000/v3 for discovery." error?03:55
eanylinI can see the placement service and endpoints created in MariaDB now03:55
*** jrobinson has joined #openstack-kolla03:56
eanylinduonghq: Yeah, seeing similar error03:56
eanylinFailed to contact the endpoint at http://keystone-internal:5000/v3 for discovery. Fallback to using that endpoint as the base url.03:57
eanylinUnable to establish connection to http://keystone-internal:5000/v3/users?: HTTPConnectionPool(host='keystone-internal', port=5000): Max retries exceeded with url: /v3/users (Caused by NewConnectionError('<03:57
duonghqeanylin, have you resolved it?03:58
eanylinif keystone internal is set to keystone public URL, I think we will not see this error03:58
eanylinduonghq: Not yet. I tested this config since fooliouno seems to face similar problems03:58
duonghqeanylin, thanks03:58
fooliouno_eanylin: I got the patch and also did a helm install for the placement service and cell003:59
eanylinI was trying to figure out the placement keystone creation failure for the most part03:59
eanylinsince sbezverk says it worked for him03:59
duonghqeanylin, ok, I'm trying to deploy only keystone and connect to it, after it's successful I'll move on03:59
eanylinfooliouno: Yeah, but not sure if you did create the placement service and keystone endpoints or not04:00
fooliouno_enaylin: which step does that. How to verify it04:00
eanylinduonghq: I think I used earlier guide with keystone internal = keystone public04:00
duonghqeanylin, I will try04:01
eanylinfooliouno: Some problems with my Openstack API calls due to the keystone-internal error04:01
eanylinI went direct to the mariadb04:01
eanylinto check04:01
fooliouno_ok04:02
eanylinduonghq: Ok04:02
duonghqeanylin, the dns resolen keystone-internal correctly04:02
eanylinduonghq: Ok, I set the kube-dns IP in my resolv.conf but wasnt able to resolve04:02
eanylinso it worked for you?04:03
duonghqeanylin, which pod are you using for test this?04:03
eanylinkolla04:03
eanylinsorry, thats the namespace04:03
duonghqI'm using busybox, it's resolve correctly04:03
eanylinah04:04
eanylinok04:04
eanylinmaybe I can try that too04:04
sdakeeanylin sbezverk hasn't run throug hthe guide - so its unclear that he has reproducibel steps - he knwos how this stuff works in detial and he uses the gate scripts as a guide04:04
fooliouno_The OS_AUTH_URL in my admin-openrc was pointing to the kolla_internal_vip_address. I changed that to my node's public IP. Now the openstack commands try to access keystone-internal and the name resolution fails.04:05
sdakei'm glad we are making really good progress though :)04:05
duonghqfooliouno_, so you stuck at some point as me04:06
fooliouno_sdake: Agree .. the new docs certainly helped speed up the process04:06
fooliouno_duonghq: looks like it :)04:06
kolla-slack<vhosakot> @sdake did not get time to test your guide today… will test tomorrow/this week for sure.04:06
sdakefooliouno_ versus guessing or reading the gate :)04:06
sdakevhsosakot I think we are going to merge something soon04:07
sdakeand then iterate on that04:07
kolla-slack<vhosakot> @sdake cool +104:07
duonghqsdake, so seem that I and fooliouno_ got same issue,04:07
eanylinsdake: I think I figured out how to get placement service/keystone endpoints created04:07
sdakeso its rendered to work from04:07
fooliouno_sdake: yep. definitely saved a lot of time for me04:07
eanylinit has got to do with sequencing04:07
eanylinon which microservice to start first04:07
kolla-slack<vhosakot> @sdake yep, good to get something merged soon, and then improve as needed.04:07
sdakeyup - this i s why it needs to be inth nova service chart04:08
eanylinsdake: Yeah, I went and read the openstack installation document04:08
eanylinfor placement04:08
kolla-slack<vhosakot> @fooliouno did you test @sdake’s guide?  if so, how far did you go?  were you able to get all OpenStack pods up?04:08
sdakeuntil that time (which is easy to get to) we can just add some waits on the various services04:08
eanylinsdake: You want me to put those steps in or?04:08
sdakeeanylin yes plz04:09
fooliouno_vhosakot: Yes .. pretty much at the end. All pods up. issue with running openstack commands.04:09
eanylinsdake: Ok. I will leave comments04:09
kolla-slack<vhosakot> @fooliouno_ nice! :slightly_smiling_face:04:09
eanylinduonghq: busy box resolves for me too04:09
*** zhubingbing has joined #openstack-kolla04:09
duonghqeanylin, nice04:09
sdakeeanylin surprised you ahve een able to replicate the deploy from the docs :)04:09
duonghqso we don't what's wrong04:09
sdakeeanylin or maybe you helped make the docs more then replicate it :)04:10
duonghq*don't know04:10
duonghqkollo container doesn't have nslookup04:10
sdakeduonghq ok - so eveyrone has the same problem then which is more specifically what?04:10
duonghqsdake, I'm finding what is cannot contact to keystone-internal04:11
sdakeduonghq the placement api is needed or nothing wil lwork04:11
sdakeok well thats a different probelm then the placement api04:11
duonghqsdake, even keystone cannot work by itself?04:11
sdakeduonghq let me asks yo uthis04:12
sdakecan you run openstack catalog list pls?04:12
duonghqsdake, I see something interesting here04:12
duonghq[sorry for raw paste]04:12
duonghq| keystone | identity | RegionOne                                    |04:12
duonghq|          |          |   public: http://192.168.122.115:5000/v3     |04:12
duonghq|          |          | RegionOne                                    |04:12
duonghq|          |          |   internal: http://keystone-internal:5000/v3 |04:12
duonghq|          |          | RegionOne                                    |04:12
duonghq|          |          |   admin: http://192.168.122.115:35357/v3     |04:12
sdakewinning04:12
sdakeok thats really god04:13
sdakegood04:13
sdakei ran int ot his problem about 3 weeks ago04:13
sdakei wish i had etter notes about what i di ot solve it04:13
sdakei knwo it involved setting a value in cloud.yaml04:13
duonghqsdake, what is the expected output?04:13
sdakethe fact that keystone spit out values means keystone works04:14
sdakecatalog list comes from keystone unauthenticated04:14
sdakee.g. port 5000 of keystone04:14
sdake35357 is the authenticated port04:14
sdakewhat error do yo usee when you run keystone endpoint list?04:15
duonghqkolla@master:~/kolla-bringup$ openstack endpoint list04:16
duonghqFailed to contact the endpoint at http://keystone-internal:5000/v3 for discovery. Fallback to using that endpoint as the base url.04:16
sdakeduonghq how did you create your keystone rc file?04:16
eanylinsdake: Updated, left the comment for nova placement04:16
sdakeeanylin cool will integreate shortly04:16
duonghqsdake, using kolla-ansible post-deploy04:16
eanylinsdake: Haha :)04:16
eanylinsdake: Cool04:17
*** schwicht has quit IRC04:17
eanylinsdake: I think we are left with the cell0 puzzle now04:17
sdakeduonghq in the tools directory is something that creates an openrc04:17
sdakecan you try it04:17
sdakewith ext as the termination04:18
sdakemoment04:18
duonghqsdake, you mean ./build_local_admin_keystonerc.sh04:18
kolla-slack<vhosakot> duonghq  can you try "openstack user list --domain default”  ?04:19
*** bmace has quit IRC04:19
sdakeduonghq yes04:19
sdakeplease run that with ext as a paraemteter04:19
sdakethen source the file it creates04:19
sdakewhich i htink is ~/keystone_adminrc04:20
sdakeor somethign like that04:20
sdakei may ahve used that method04:20
sdakerather hten post-deploy04:20
duonghqsdake,  you mean with arg xext?04:20
sdakejust ext04:21
sdakexext==x the x tests for an empty value04:21
sdakeshell ftl04:21
duonghqah,04:21
duonghqsdake, voila04:22
sdakedid that work?04:22
duonghqyes04:22
sdakenice!04:22
sdakewho else was suffering that problem?04:22
sdakeplase leave a comment in the docs review04:22
sdakeI will address that now04:22
fooliouno_eanylin: saw your comment on order of placement microservices. Should all of them be run after nova-control and nova-compute?04:22
eanylinfooliouno, I havent really test the order but I think so, its probably ok to run after that04:23
eanylinI saw in some forum that they manually do it to upgrade their environment04:24
eanylinduonghq: I think I found a way to get round that. Not very elegant though04:24
fooliouno_ok .. we were not running all of them earlier.04:24
sdakeeanylin souds like he got around it04:24
eanylinfooliouno: I wanted to run those but I keep getting keystone endpoint create fails04:25
sdakeand it is elegant04:25
eanylinso was investigating why it was not working04:25
eanylincool, so it worked for duonghq04:25
sdakeyup running that shell script and sourcing keystone_adminrc are what i did - and I docuemnted the wrong thing04:26
eanylinoh04:26
eanylinI went and use nslookup to find that IP04:26
duonghqeanylin, sdake the "new" rc file use admin endpoint port04:26
duonghqwith a couple of extra settings04:27
sdakeyup crafting an openrc file is a little difficult04:27
duonghqbut PS1=... is redundant04:27
sdakepost-deploy used to work fine04:27
sdakenot sure why it doesnt' work now04:27
sdakedoesn't matter as we are getting rid of all that04:27
eanylinso post deploy is not the right one to use then?04:27
sdake(in kolla-kubernetes)04:28
duonghqeanylin, for kolla-k8s, seem that04:28
sdakea python implementation will probably hit for that ile04:28
duonghqthe bigest different is the port04:28
eanylinduonghq: I see04:28
sdakeduonghq mind pasting both versions04:28
duonghqsure04:28
eanylinduonghq sdake: The new method works04:29
duonghqsdake,  http://paste.openstack.org/show/604608/04:29
eanylinis PS1 redundant?04:30
duonghqeanylin, yes04:30
*** dixiaoli has quit IRC04:30
eanylinduonghq: Seems so04:30
eanylinI unset that04:31
eanylinit works too04:31
duonghqeanylin, yup, it does not have any business here04:31
eanylinagree04:31
eanylinso guess just need to update the guide to run this command instead of the old one04:31
sdakewonder what this is: build_local_admin_keystonerc.sh04:31
kolla-slack<vhosakot> duonghq  did the keystone CLI work?04:32
sdakeexport OS_INTERFACE=internal04:32
sdakerather04:32
duonghqvhosakot: I cannot install keystone CLI04:32
duonghqbut openstack user list work04:33
sdakeduonghq use openstack client04:33
fooliouno_duonghq: Whats the IP in your OS_AUTH_URL pointing to04:33
sdakekeystone client is busted04:33
duonghqsdake, sure04:33
duonghqfooliouno_, I'm using AIO, so it's the node ip04:33
kolla-slack<vhosakot> I meant, sorry openstack CLI, I know keystone is deprecated :slightly_smiling_face:04:33
kolla-slack<vhosakot> I meant keystone CLI is deprecated :slightly_smiling_face:04:34
duonghqvhosakot: we know what you meant :P04:34
eanylinopenstack service list works too04:34
eanylinshould be good04:34
duonghqglance works fine04:35
kolla-slack<vhosakot> cool, keystone auth works! :slightly_smiling_face:04:35
duonghqvhosakot: nice04:36
kolla-slack<vhosakot> no, I have not tried it yet :slightly_smiling_face:  I was happy for you guys :slightly_smiling_face:04:36
duonghq:)04:37
eanylin:)04:37
duonghqeanylin, you are Anthony Lin?04:37
eanylinduonghq: Yes, I am04:37
duonghqeanylin, placement api microservice should be installed after all other service?04:38
eanylinhmm, actually I havent tested, so I am not sure too04:38
eanylinI know how to get it into the mariaDB now04:38
eanylinit wasnt getting created earlier04:38
duonghqeanylin, ya,04:38
eanylinif I dont follow certain sequencing04:38
fooliouno_sdake: enaylin: duonghq: The new keystonerc works for me as well! :)04:39
duonghqfooliouno_, great04:39
*** yuanying has joined #openstack-kolla04:39
eanylinfooliouno: Cool :)04:39
*** unicell has joined #openstack-kolla04:39
eanylinduonghq: I think we can try the sequence of creation of the nova services04:40
duonghqeanylin, can you explain?04:40
eanylinto see if it makes any difference04:40
duonghqI'm installing neutron04:40
duonghqhead to nova now04:40
eanylinlike create nova-control,compute, then placement and then cell004:41
eanylinI think that might be the right sequence04:41
eanylinsince cell0 depends on placement04:41
eanylinI think04:41
fooliouno_Any idea how to create a public endpoint for Horizon so that it is accessible directly via browser from the outside04:41
*** gkadam has joined #openstack-kolla04:42
eanylinfooliouno: We were trying port forward earlier04:42
eanylinlike port forward port 80 of horizon pod04:42
fooliouno_ok04:42
eanylinbut I think there might be some problems with it, saw some issues earlier04:42
sdakethere is an easier way to make the port external04:43
duonghqfooliouno_, just one setting in cloud.yaml04:43
*** bmace has joined #openstack-kolla04:44
fooliouno_duonghq: whats the setting04:44
duonghqfooliouno_, you can see helm/all_values.yaml04:44
duonghqit's default one04:44
fooliouno_Let me check04:44
duonghqand cloud.yaml will override that04:44
eanylinoh04:45
eanylinso just need to make it external and it will override?04:45
sdakeeanylin looking for optoin moment04:46
duonghqone more thing: kubernetes does not allow we use node_port with port is 8004:46
duonghqby default in kubeadm04:46
fooliouno_port_external should be set to true?04:46
eanylinI did this04:47
duonghqfooliouno_, yup04:47
eanylin[root@kube03 ~]# kubectl port-forward horizon-3565390796-4w349 -n kolla 8004:47
eanylinForwarding from 127.0.0.1:80 -> 8004:47
eanylinForwarding from [::1]:80 -> 8004:47
eanylinHandling connection for 8004:47
eanylinHandling connection for 8004:47
eanylinHandling connection for 8004:47
eanylinso you can see it will also work04:47
*** shashank_t_ has joined #openstack-kolla04:47
sdakeeanylin thats too complicated :) - there is an eaiser way04:47
sdakeglance-api-svc:04:47
sdake        global:04:47
sdake            kolla:04:47
sdake                glance:04:47
sdake                    api:04:47
sdake                        all:04:47
sdake                            port: 929204:47
sdake                            node_port: 929204:47
sdake                            port_external: false04:47
sdake                            node_port_enabled: false04:47
sdakein cloud.yaml set port-external: true04:48
eanylinsdake: Ok, let me check04:48
sdakeport-external04:48
sdakeport_external04:48
sdakegah - keyboard04:48
sdakedont do it in all_values.yaml - instead cloud.yaml overrides thsoe defaults04:48
fooliouno_yup .. I was looking there for the field.04:48
sdakeall_values.yaml contains all fields04:49
sdakecloud.yaml is a subset of overrides from the defaults04:49
duonghqneed leaving now, BRB04:49
sdakelater duonghq04:49
spsuryasdake: can you please add me too as Co-Author  in this preparation https://review.openstack.org/#/c/447356/  ?04:49
fooliouno_later duonghq. Thansk for the help04:49
sdakespsurya sure04:49
eanylinlater duonghq04:50
spsuryasdake: thanks04:50
eanylin    horizon:04:50
eanylin      all:04:50
eanylin        node_port: 8004:50
eanylin        node_port_enabled: false04:50
eanylin        port: 8004:50
eanylin        port_external: true04:50
eanylinshould work?04:50
fooliouno_I think so. I made the same change, but will try it tomorrow.04:52
sdakespsurya what is your email address - gerrit isn't popping it up for me for some reason04:52
spsuryasdake: surya.singh@nectechnologies.in04:53
fooliouno_I'm gonna call it a night. Thanks everyone!04:53
spsuryaplease use this04:54
spsuryasdake:04:54
eanylinfooliouno: That is the right config04:54
eanylinI just tested04:54
fooliouno_you are fast :)04:55
eanylintake note of the indentation though, for the yaml file04:55
eanylin:)04:55
eanylinI delete my horizon and create again :D04:55
eanylinsdake: I am going to update the cloud.yaml to include this for Horizon04:57
sdakeeanylin i hink that would be better then what i mentioned04:58
sdakeeanylin yo ushouldn't have to override node_port or port por node_port-enabled04:58
sdakeshoudl be able to do horizon:04:59
sdake     all:04:59
sdake    port_external: true04:59
sdakewe are after a minimal working cloud.yaml04:59
eanylinsdake: Ok05:00
eanylinmakes sense05:00
eanylinrather than repeating05:01
eanylinsince the rest are default05:01
*** iceyao has joined #openstack-kolla05:02
eanylinupdated05:03
spsuryasdake: hope you got surya.singh@nectechnologies.in05:03
sdakespsurya yup - surya is your first name?05:04
*** obre_ has joined #openstack-kolla05:06
*** obre has quit IRC05:07
*** kwazar has quit IRC05:07
*** markus-k has quit IRC05:07
*** kwazar_ has joined #openstack-kolla05:07
*** markus-k_ has joined #openstack-kolla05:07
spsuryasdake: Surya Prakash is the first name05:10
sdakebuild_example_yaml.py doesn't work05:11
spsuryaand Singh is the last05:11
sdakespsurya thanks05:11
sdakejust want to get hte name correct in the review05:11
spsuryasdake: nic <spsurya>  is also fine :)05:12
sdakespsurya has to be formatted correctly for gerrit i think05:13
*** iceyao has quit IRC05:13
sdakewelli  just screewed up my work on the review05:14
sdakei think i am headed to bed05:14
sdakeperhaps people can add their comments05:14
sdakeand I'll address in the morning05:14
*** iceyao has joined #openstack-kolla05:14
spsuryasdake: sure05:14
spsuryaBTW night05:14
*** iceyao has quit IRC05:15
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Deployment Guide Documentation  https://review.openstack.org/44735605:22
*** adrian_otto has joined #openstack-kolla05:26
*** dixiaoli has joined #openstack-kolla05:31
*** dixiaoli has quit IRC05:34
*** sayantan_ has joined #openstack-kolla05:46
*** shashank_t_ has quit IRC05:54
*** zhurong has quit IRC05:54
*** shashank_t_ has joined #openstack-kolla05:55
*** dixiaoli has joined #openstack-kolla05:58
*** shashank_t_ has quit IRC05:59
*** jrobinson has quit IRC06:01
openstackgerritSurya Prakash (spsurya) proposed openstack/kolla-kubernetes master: Deployment Guide Documentation  https://review.openstack.org/44735606:02
openstackgerritZeyu Zhu proposed openstack/kolla-kubernetes master: Add a script to generate configuration  https://review.openstack.org/45119906:12
*** jaosorior has joined #openstack-kolla06:14
eanylinsdake: Left a comment on create-cell-job, which I think is related to cell0.  Take a look when you are up in the morning. Thanks.06:14
*** Serlex has joined #openstack-kolla06:21
*** pcaruana has joined #openstack-kolla06:24
*** tovin07 has joined #openstack-kolla06:25
*** caowei has quit IRC06:26
*** caowei has joined #openstack-kolla06:27
*** targon has joined #openstack-kolla06:28
*** bjolo has joined #openstack-kolla06:41
Jeffrey4lsbezverk, around?06:46
*** oanson has quit IRC06:48
openstackgerritchenyingnan proposed openstack/kolla-ansible master: Allow nfs, rpcbind and mountd service on server firewall  https://review.openstack.org/45121006:49
duonghqanybody got Target WSGI script not found or unable to stat: /usr/share/openstack-dashboard in horizon (kolla-k8s)06:49
*** matrohon has joined #openstack-kolla06:55
*** athomas has joined #openstack-kolla06:57
*** matrohon has quit IRC07:01
openstackgerritSurya Prakash (spsurya) proposed openstack/kolla-kubernetes master: Deployment Guide Documentation  https://review.openstack.org/44735607:06
*** sayantan_ has quit IRC07:18
*** sayantan_ has joined #openstack-kolla07:19
*** sayantan_ has quit IRC07:23
duonghqspsurya, ping07:23
spsuryaduonghq: hi..07:24
duonghqabout kobernetes-deploymen guide07:24
duonghqwhy do you remove line 10207:24
duonghqI mean the slash is redundant, not whole comment07:25
duonghq*command07:25
*** jmccarthy has joined #openstack-kolla07:25
spsuryaduonghq: ohh07:25
spsuryaduonghq: please update that :(07:25
spsuryaduonghq: by mistake *dd* done that :) i guess07:27
spsuryai was removing the below line once07:28
duonghqspsurya, I will update07:28
spsuryaduonghq:07:29
spsuryathanks07:29
spsuryameeting time BRB07:29
openstackgerritDuong Ha-Quang proposed openstack/kolla-kubernetes master: Deployment Guide Documentation  https://review.openstack.org/44735607:30
*** britthouser has joined #openstack-kolla07:31
*** britthouser7 has quit IRC07:31
*** zhurong has joined #openstack-kolla07:34
*** tovin07 has quit IRC07:45
hrwmorning07:54
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: make yum fail on missing packages  https://review.openstack.org/44908607:56
hrwrebased on master07:57
bjolomorning07:57
spsuryahrw: bjolo morning07:58
spsuryahrw: https://review.openstack.org/#/c/440634/   ask some core for this review07:59
spsuryahrw:  as i am not sure about this07:59
hrwspsurya: this one is kind of problematic to me.07:59
spsuryahrw: yes08:00
hrwspsurya: I am sure that previous state was wrong (ceph-fuse in ceph-base) but is my suggestion right? do not know08:00
*** Mech422 has quit IRC08:01
spsuryahrw: not so sure :(08:01
hrwgnocchi will get one patch later today. but have to check some logs first08:02
spsuryahrw: ok08:03
*** shardy has joined #openstack-kolla08:03
hrwgnocchi-base/source takes huge amount of time on !x86-6408:05
*** nathharp has joined #openstack-kolla08:07
*** gfidente has joined #openstack-kolla08:08
spsuryahrw: if we see the existing x86 executables run with no compatibility or performance penalties08:09
spsuryathats include in packages itself08:10
duonghqI'm sorry but who is "Marcin Juszkiewicz"?08:10
duonghqin irc?08:10
spsuryahrw is08:10
spsuryaAFAIK08:10
spsuryahrw: am i right ?08:11
hrwduonghq: me08:11
duonghqhrw, so I remember correctly,08:12
duonghqabout this ps: https://review.openstack.org/#/c/449086/08:12
hrw:)08:12
hrwduonghq: yum one, yes?08:12
duonghqyou mean if we miss some package on x86 arch, it still fine?08:12
hrwduonghq: no, I mean that on x86 there are no missing ones08:12
hrwduonghq: those missing ones got cought and fixes got merged last week08:13
duonghqhrw, technically, nothing is impossible,08:13
hrwduonghq: sure.08:13
duonghqbtw, you got it much more time in non-x86 arch?08:13
*** dixiaoli has quit IRC08:13
hrwduonghq: huge amount of times08:13
duonghqcan you explain the reason?08:14
hrwduonghq: most of my patches are "on aarch64 or ppc64le this package is missing"08:14
duonghqhrw, due to fail to connect to repository? or what?08:14
hrwduonghq: for centos/x86-64 kolla is using several external repositories. None of them provide packages for aarch64 nor ppc64le08:14
hrwduonghq: so for example no grafana, no fluentd08:15
duonghqso, in poor connection env, there is possible that I cannot contact to any repository08:15
duonghqi.e. I face same issue as your08:15
hrwduonghq: sure, but then your build fail on "yum: cannot connect to repo"08:15
hrwduonghq: where on !x86-64 I do not even use those repos08:16
*** adrian_otto has quit IRC08:16
duonghqit do not output you cannot connect to these repos08:16
duonghqinteresting08:17
hrwduonghq: it does. I had several CI failures last week due to grafana repo being nonaccessible08:17
duonghqwait08:17
hrwand CI runs on x86-6408:17
duonghqwithout your patch, it still fails or not?08:18
duonghqsorry, but I lost some thought08:18
hrwit fails without08:18
*** dixiaoli has joined #openstack-kolla08:18
duonghqso, with your patch, it still fails?08:19
hrwwith or without it fails when repo is unaccessible08:19
hrwbecause it happens before yum starts to check what it has to download/install08:19
duonghqah, but w/o it, it does not fail if the repo still accessible but not contain the package?08:20
*** pbandark has joined #openstack-kolla08:20
hrwduonghq: other example: I do build of aodh-base for centos/binary. There is no openstack-aodh-common package for aarch64 so without my patch I get aodh-base image which lacks aodh. with my patch I get build failure08:20
hrwduonghq: if repo is not accessible then my patch does not change anything08:20
spsuryaduonghq: +108:21
spsuryawithout it should not be fail08:21
hrwsorry, I got lost ;d08:21
hrw10:20 < duonghq> ah, but w/o it, it does not fail if the repo still accessible but not contain the package?08:21
hrwit not fail08:21
hrwjust be useless08:21
duonghqgot it08:22
duonghqinteresting case :)08:22
hrwit may fail if Dockerfile.j2 plays with files from package and they are not present but that's only some images08:22
duonghqthank for explain it, hrw08:22
hrwduonghq: yw08:22
hrwduonghq: can you mark it W+1 then?08:22
duonghqhrw, gate is broking again :(08:23
*** mgoddard has joined #openstack-kolla08:23
spsuryahrw: it may fail if Dockerfile.j2 plays with files from package and they are not present but that's only some images08:23
spsuryagot you hrw08:23
hrwspsurya: image isntalls package. package contains /usr/share/package/package/py and Dockerfile.j2 seds it. no file == failure08:24
* duonghq tire with gate :(08:24
hrwduonghq: I meant +1 for workflow :P08:24
duonghqhrw, I know08:25
hrw:)08:25
duonghqbut I need all gates are green :(08:25
hrwduonghq: gate... I agree that it is pain08:25
spsuryahrw: roger08:25
duonghqwish that I can merge gate result from different run08:25
duonghqand you'll have all green gate08:25
hrwbut that would be cheating08:26
*** matrohon has joined #openstack-kolla08:27
duonghqnot sure, I don't think union operator cheat anybody08:27
duonghqwe have some code08:27
duonghq*same08:27
duonghqand have gate green for x%08:27
duonghqfor 12 long running gate08:28
*** matrohon_ has joined #openstack-kolla08:28
hrwduonghq: sure. this time.08:29
duonghqwe still recheck again until gate is green, so if union is a cheat, recheck is equivalent08:29
hrwduonghq: but patch1 can be right on d.m.y and wrong day later because patch2 got merged in meantime08:29
duonghqgive gate is independent on others08:29
duonghqlet me rephrase, given same code, no rebase, no edit, ....08:30
*** gkadam is now known as gkadam-mtg08:30
duonghqit should be consistence from time to time08:30
duonghqbut it doesn't08:30
duonghqso we recheck08:30
hrwyep08:30
duonghqand hoping for it go green with some chance08:30
duonghqit's equivalent of trust to merging all test result from multiple run08:31
duonghqgiven exactly same code08:31
*** matrohon has quit IRC08:31
*** yuanying has quit IRC08:31
spsuryaagreed with duonghq08:32
duonghqand bad news is your ps get red on deploy-ubuntu-source-ubuntu-xenial08:32
*** gkadam_ has joined #openstack-kolla08:32
duonghqand we need waiting half hour before next recheck08:33
duonghqhrw, something wrong with haproxy log in gate of your ps08:35
duonghqnot good08:35
*** gkadam-mtg has quit IRC08:35
hrwduonghq: in which patch?08:35
duonghqthe yum patch08:35
duonghqhrw, here http://logs.openstack.org/86/449086/5/check/gate-kolla-dsvm-deploy-ubuntu-source-ubuntu-xenial-nv/a774ee1/08:36
duonghqdoesn't related to your change08:36
hrwINFO:kolla.image.build.haproxy:adduser: The user `haproxy' already exists. Exiting.08:36
hrwthat's package bug08:37
hrwbut image got built anyway08:37
duonghqhrw, http://logs.openstack.org/86/449086/5/check/gate-kolla-dsvm-deploy-ubuntu-source-ubuntu-xenial-nv/a774ee1/console.html#_2017-03-29_08_14_57_18815908:38
duonghqthis failed08:38
hrwah08:38
duonghqhrw, you let all of your ps related to previous work, so nobody can get anything merge soon :+08:44
duonghq:(08:44
hrwduonghq: they use base_arch variable which is implemented in 1st patch08:46
*** dixiaoli_ has joined #openstack-kolla08:46
duonghqhrw, understand :(08:47
duonghqcan you raise it in today meeting,08:47
hrwyep08:47
duonghqnice08:47
*** dixiaoli has quit IRC08:48
hrwaddd to agenda08:49
spsuryaduonghq: In your opinion08:52
spsuryawhat all left for this https://review.openstack.org/#/c/425446/08:53
spsuryaI guess we can complete this in pike right ?08:53
spsuryaduonghq: it would be great if you will provide your thought on this08:54
*** yangyapeng has joined #openstack-kolla08:57
hrwduonghq: will it help if I split https://review.openstack.org/#/c/430940/ (1st patch in series) into two parts? One will just add base_arch variable into code. So nothing will change for anyone. but it will be simple patch which enable several patches to mergepath.08:57
openstackgerritBertrand Lallau proposed openstack/kolla-ansible master: Remove ml2_conf.ini file from some Neutron agents  https://review.openstack.org/44799208:57
hrwduonghq: and no other architectures added in it08:58
duonghqhrw, should bring the idea to the meeting, but it's nice IMO09:00
duonghqspsurya, gate green, so it should be good, even we didn't test it well09:00
*** yangyape_ has joined #openstack-kolla09:01
duonghqwe must revised it latter09:01
duonghq*revise09:01
duonghqspsurya, I'm not sure, need more dicussion09:02
spsuryaduonghq: i thing we should mark workflow -1 or commit tile as WIP so that we can finish testing upto a lavel09:02
hrwduonghq: https://paste.fedoraproject.org/paste/WuvJvmJCSUaFtx2b8OFJkF5M1UNdIGYhyRLivL9gydE=/ is whole base_arch patch09:02
spsuryaduonghq: then we can thing about its review and merging09:03
spsuryaas this is essential one09:03
*** yangyapeng has quit IRC09:04
duonghqspsurya, we should bring it to meeting09:04
*** egonzalez has joined #openstack-kolla09:04
spsuryaduonghq: we already did that 2 times IIRC09:05
spsurya:)09:05
duonghqspsurya, sorry, my thought get jamming now09:05
duonghq:(09:05
duonghqspsurya, yes, and many CRs agree that it can be merged09:05
spsuryamay be egonzalez can confirm that09:06
*** manheim has joined #openstack-kolla09:06
spsuryaduonghq: then need review09:06
egonzalezspsurya about?09:06
spsuryahttps://review.openstack.org/#/c/425446/  egonzalez about this one09:06
*** manheim has quit IRC09:07
*** manheim has joined #openstack-kolla09:07
spsuryaegonzalez: do we need to test it up to a level09:07
duonghqmorning egonzalez09:08
spsuryaegonzalez: at least the basic one which is mentioned https://etherpad.openstack.org/p/ks-rolling-upgrade-test09:08
egonzalezspsurya yep, there was a time since i tested the last time, some things may changed in keystone side09:09
hrwegonzalez: https://paste.fedoraproject.org/paste/WuvJvmJCSUaFtx2b8OFJkF5M1UNdIGYhyRLivL9gydE=/ - would you +2 it?09:09
egonzalezhrw wrong  link tho09:10
hrwegonzalez: it's raw patch so far09:10
spsuryaegonzalez: ok09:11
hrwegonzalez: I am thinking of doing my !x86-64 patchset in other way. to make it easier to merge09:11
hrwegonzalez: base_arch change does not change anything but allows to get several of my patches into own patchsets making main one smaller09:12
spsuryaegonzalez: i will check and will comment on the patch set, if any changes has been done from keystone side09:12
egonzalezhrw i think that  is the good way, otherwise changes will be failing with merge conflict forever. small non breaking changes is the  good way imo09:13
hrw:)09:13
hrwwill then sent it right now and then rebase on top09:13
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: add base_arch variable for future non-x86 work  https://review.openstack.org/45126809:18
hrwegonzalez, duonghq: can you review that and +2?09:18
*** gkadam_ has quit IRC09:20
*** rmart04 has joined #openstack-kolla09:21
egonzalezhrw not sure if some tests for base arch should be included in test_build, not really familiar with code coverage09:22
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: add base_arch variable for future non-x86 work  https://review.openstack.org/45126809:25
hrwfound bug ;(09:25
hrwsolved09:25
pomacanyone that can help me wrap my head around ansible - i've enable debug and *_KEEP_REMOTE_* environmental variables, all of this leads me to belive that a error is local, in the generation of ansiballz or so - https://pastebin.com/yx81BAkU <- execution trace - the result doesn't seem to ever be stored on the remote host and from what i gather all data is available09:26
pomacThe failure is: "msg": "'dict object' has no attribute u'ansible_bond0'"09:26
egonzalezpomac if you do ansible -m setup -i inventory host ? is bond0 in the list of ansible interfaces?09:27
pomacIn the debug promt: p vars['hostvars'][str(host)]['ansible_bond0'] works09:27
pomacyes09:27
pomacthis is only when it seems to generate the nova config - the merge pass09:27
pomacegonzalez: trying to figure out where in ansible this happens is.... hard... =)09:28
pomacp vars['hostvars'][str(host)]['ansible_bond0']['ipv4']['address'] also works09:29
pomac(lunch, afk)09:29
*** sambetts|afk is now known as sambetts09:30
*** gkadam_ has joined #openstack-kolla09:31
*** gkadam_ is now known as gkadam09:35
*** cu5 has joined #openstack-kolla09:35
openstackgerritshaofeng cheng proposed openstack/kolla-ansible master: Add networking with vmware vsphere  https://review.openstack.org/45128909:36
*** yuanying has joined #openstack-kolla09:39
*** manheim has quit IRC09:41
*** yuanying has quit IRC09:44
*** caoyuan has quit IRC09:45
*** yuanying has joined #openstack-kolla09:47
*** duonghq has quit IRC09:48
hawihi. what is needed in order to get nice grafana dashboard about kolla installation? i'm just trying to figure out, what to use for it, collectd, influxdb, elasticsearch, telegraf?09:52
*** yingjun has quit IRC09:54
openstackgerritSurya Prakash (spsurya) proposed openstack/kolla-kubernetes master: py34 not in use for the deliverable  https://review.openstack.org/45130009:56
*** matrohon_ has quit IRC09:57
*** yangyape_ has quit IRC09:59
*** manheim has joined #openstack-kolla10:00
*** yangyapeng has joined #openstack-kolla10:00
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: debian: enable all images enabled for Ubuntu  https://review.openstack.org/43278710:07
hrwanother change extracted from huge patchset into separate one10:07
egonzalezhrw added a comment https://review.openstack.org/#/c/449086/10:15
egonzalezfeatures cannot be backported to stable branchs, and partially-implements in commit is saying that is a feature10:16
hrwok10:18
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: make yum fail on missing packages  https://review.openstack.org/44908610:18
hrwegonzalez: dropped it then10:18
* hrw -> splitting patchset into parts10:20
egonzalezhrw thanks, atm were are applying for stable policy, don't want to risk that10:20
hrwsure10:20
hrwegonzalez: I care more about getting patch in that will it be linked to blueprint ;d10:20
*** tovin07_ has quit IRC10:20
hrws/that/than10:20
*** cuongnv has quit IRC10:21
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: ironic-pxe: handle non-x86 architectures  https://review.openstack.org/43481710:26
openstackgerritSurya Prakash (spsurya) proposed openstack/kolla-kubernetes master: Deployment Guide Documentation  https://review.openstack.org/44735610:27
*** pbourke has quit IRC10:27
*** pbourke has joined #openstack-kolla10:28
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: kubernetes: disable for architectures other than x86-64  https://review.openstack.org/44563810:29
*** caowei has quit IRC10:30
pbourkeegonzalez: Daviey Jeffrey4l zhubingbing: hi can some of you please review https://review.openstack.org/#/c/438030/10:30
pbourkethanks10:30
zhubingbingok10:30
*** MarginHu has quit IRC10:32
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: use mariadb-devel for CentOS and !x86-64  https://review.openstack.org/44483210:37
hrwpbourke: what do you think about https://review.openstack.org/451268 one?10:38
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: nova-compute: handle rtslib(-fb) for debian-binary builds too  https://review.openstack.org/43594110:39
pbourkehrw: "...requires checking again and again after each change/merge/refresh"10:40
pbourkehrw: what do you mean?10:40
hrwpbourke: each merge conflict (~every second day) I have to push whole patchset and then anyone who did review gets flooded with 'please review patch which did not changed since last time but got reuploaded'10:42
hrwpbourke: and patchset grows and grows10:42
pbourkeso how does this solve that?10:43
hrwpbourke: with base_arch in master most of patches are separate from each other10:43
pbourkeok10:43
hrwpbourke: they do not need non-x86 support merged to work fine on x86-6410:44
hrwand once non-x86 get merged they magically work on non-x86 too10:44
pbourkehrw: one small comment on the usage msg10:45
hrwpbourke: I am open to comments10:45
hrwwill take care10:46
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: ceph: move ceph-fuse package to cephfs-fuse image  https://review.openstack.org/44063410:46
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: add base_arch variable for future non-x86 work  https://review.openstack.org/45126810:49
*** egonzalez has quit IRC10:50
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: openstack-base: Percona-Server is x86-64 only  https://review.openstack.org/44996510:52
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: nova-compute: enable Debian/source build  https://review.openstack.org/45080510:54
hrwok, now most of patches should be not related to each other10:55
manheimhi, i have a question regarding multinode and docker registry https://docs.openstack.org/developer/kolla/newton/multinode.html10:56
manheimthe registry can be on any host on my lan, right?10:56
manheimthen on the deployer node I specify the IP of my registry10:56
*** zhubingbing has quit IRC10:56
*** qiliang27 has quit IRC10:58
*** qiliang27 has joined #openstack-kolla10:58
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: mariadb: handle lack of external repos on non x86  https://review.openstack.org/44483211:03
*** yangyapeng has quit IRC11:05
*** matrohon_ has joined #openstack-kolla11:07
pbourkehrw: just fyi, there's really no need to track your different patchsets in the commit msg11:07
hrwpbourke: you mean my comment in 444832 one?11:08
hrwpbourke: I prefer to mention it to not loose track on my own too11:09
*** nathharp has quit IRC11:09
hrw13:09 hrw@gossamer:kolla$ git branch |grep to-merge|wc -l11:09
hrw1911:09
* hrw off for ~1h11:11
*** thomas_oneill is now known as thomas-oneill11:16
*** zhurong has quit IRC11:34
*** matrohon_ has quit IRC11:37
*** rwallner has joined #openstack-kolla11:40
*** gkadam is now known as gkadam-afk11:47
*** blallau has joined #openstack-kolla11:48
*** dixiaoli_ has quit IRC11:49
*** zhubingbing has joined #openstack-kolla11:49
*** dixiaoli has joined #openstack-kolla11:50
*** pbandark is now known as pbandark`brb11:52
*** rhallisey has joined #openstack-kolla11:54
*** dixiaoli has quit IRC11:54
openstackgerritMick Thompson proposed openstack/kolla-ansible master: Update keystone haproxy config to balance based on source ip  https://review.openstack.org/45135811:55
*** athomas has quit IRC11:57
*** Pavo has joined #openstack-kolla11:58
*** athomas has joined #openstack-kolla12:05
*** cu5 has quit IRC12:07
*** egonzalez has joined #openstack-kolla12:07
sdakeJeffrey4l around?12:09
mgoddardhi pbourke, would you be able to re-review and (ideally) restore your +2 on https://review.openstack.org/#/c/436462/?12:09
sdakemorning flks12:09
sdakesup mgoddard12:09
mgoddardhey sdake12:10
sdakemgoddard there is a patch in the rview queue related to ironic binary12:10
sdakeand sudoers file12:10
egonzalezmorning sdake12:10
sdakesup egonzalez12:10
mgoddardsdake: correct. would be nice to move it along12:10
sdakemgoddard i thought you fixed all that?12:11
sdakemgoddard have a link?12:11
mgoddardsdake: assume you're referring to https://review.openstack.org/441605 and https://review.openstack.org/445587?12:11
sdakemgoddard no something else12:12
sdakemgoddard lemme wake up alittle its 5am - and i'll find a link if possible12:13
mgoddardsdake: sure, let me know12:13
*** gkadam-afk is now known as gkadam12:14
mgoddardsdake: you referring to this: https://review.openstack.org/#/c/448977?12:14
*** signed8bit_Zzz is now known as signed8b_12:14
sdakeno12:16
*** kwazar_ is now known as kwazar12:22
*** schwicht has joined #openstack-kolla12:29
sdakemgoddard https://review.openstack.org/#/c/450274/12:31
hrwre12:33
sdakemgoddard does that need a backport?12:33
*** schwicht has quit IRC12:35
*** bjolo has quit IRC12:35
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: mariadb: handle lack of external repos on non x86  https://review.openstack.org/44483212:37
*** admin0 has joined #openstack-kolla12:37
*** schwicht has joined #openstack-kolla12:38
mnaserpbourke could you remove your -1 as it was shown that this was an actual cherry picked change12:39
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: ironic-pxe: handle non-x86 architectures  https://review.openstack.org/43481712:39
egonzalezmliima around12:39
*** krtaylor has quit IRC12:40
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: kubernetes: disable for architectures other than x86-64  https://review.openstack.org/44563812:40
hrwmliima: thx for Depends-On reminder12:41
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: openstack-base: Percona-Server is x86-64 only  https://review.openstack.org/44996512:41
hrwcan someone upvote https://review.openstack.org/#/c/449086/ (yum)?12:43
openstackgerritBertrand Lallau proposed openstack/kolla master: Remove Ubuntu Trove postinst script hack  https://review.openstack.org/45137812:44
*** shashank_t_ has joined #openstack-kolla12:45
*** dave-mccowan has joined #openstack-kolla12:46
Jeffrey4lsdake, yep.12:47
*** rwallner has quit IRC12:47
sdakeJeffrey4l did inc0 talk to yo uabout todays's meeting?12:47
*** signed8b_ is now known as signed8bit_Zzz12:47
*** rwellum has joined #openstack-kolla12:48
*** Deys has joined #openstack-kolla12:48
Jeffrey4lsdake, yep. i will hold today's meeting.12:48
*** lamt has joined #openstack-kolla12:48
*** matrohon_ has joined #openstack-kolla12:48
*** shashank_t_ has quit IRC12:50
*** lamt has quit IRC12:52
mliimahi egonzalez12:52
sbezverkgood morning, new kube 1.6.0 bits broke our gate. I filed the issue with kubernetes developers: https://github.com/kubernetes/kubernetes/issues/4380512:52
mliimahrw, :)12:53
*** gkadam has quit IRC12:53
sdakesbezverk not surprising12:54
sdakesbezverk a million things could have changed12:54
sdakesbezverk which versoin of docker-engine is being used?  12.6?12:55
sdakeI am pretty sure  Iwant with 1.6.0-rc1 with kubeadm and it did deploy12:55
*** schwicht has quit IRC12:55
hrwhttps://review.openstack.org/#/c/451268/ (base_arch variable) - any comments (or upvotes)?12:56
*** goldyfruit has joined #openstack-kolla12:59
*** lamt has joined #openstack-kolla13:00
*** bswartz has left #openstack-kolla13:00
openstackgerritMerged openstack/kolla master: Make keystone_user module fully shade  https://review.openstack.org/44801413:04
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Fixing 1.6.0 kubeadm init getting stuck  https://review.openstack.org/45139113:10
*** mattmceuen has joined #openstack-kolla13:10
*** rwallner has joined #openstack-kolla13:12
*** matrohon_ has quit IRC13:13
*** JoseMello has joined #openstack-kolla13:14
*** Deys has quit IRC13:14
rwellumsdake: sbezverk argh I was seeing that last night following the guide - I was sure I was doing something wrong...13:16
*** lamt has quit IRC13:17
mnaserlol13:18
mnaserbluebox cloud has a host running on 169.254.169.1013:18
*** schwicht has joined #openstack-kolla13:18
mnaserso prechecks cause job failures13:18
sbezverkrwellum: nope, and that fix is not enough, other things got broken too13:18
mnaserhttp://logs.openstack.org/03/450203/2/check/gate-kolla-dsvm-deploy-centos-binary-centos-7-nv/f917f47/console.html13:18
mnaserfun times13:19
*** erlon has joined #openstack-kolla13:20
*** rwallner has quit IRC13:21
rwellumsbezverk: 😢13:22
*** rwallner has joined #openstack-kolla13:23
*** zhurong has joined #openstack-kolla13:23
hawiis this something usua: The content of source file(/var/lib/kolla/config_files/galera.cnf) and dest file(/etc/mysql/my.cnf) are not equal13:25
hawii cant start galera any more13:25
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Fixing 1.6.0 kubeadm init getting stuck  https://review.openstack.org/45139113:26
*** Deys has joined #openstack-kolla13:28
*** signed8bit_Zzz is now known as signed8b_13:28
*** jmontleon has quit IRC13:29
*** jmontleon has joined #openstack-kolla13:29
*** mliima_ has joined #openstack-kolla13:31
sdakerwellum note people yesterday evening reported the guide got them operational13:31
*** esharao has joined #openstack-kolla13:31
sdakerwellum although it was about10pm MST so i hit the rack13:31
openstackgerritMauricio Lima proposed openstack/kolla-ansible master: Pg_num should be configurable  https://review.openstack.org/45079713:31
openstackgerritOpenStack Proposal Bot proposed openstack/kolla master: Updated from global requirements  https://review.openstack.org/45101613:33
*** eanylin has quit IRC13:34
*** mliima has quit IRC13:34
*** zhubingbing has quit IRC13:37
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible master: Fix ceph log location  https://review.openstack.org/45140713:37
*** jmontleon has quit IRC13:38
*** jaosorior has quit IRC13:38
*** jtriley has joined #openstack-kolla13:39
sdakesbezverk looking in the yum repos, i dont see a kubernetes 1.6.013:39
*** dmsimard is now known as dmsimard|afk13:39
sdakesbezverk although clearly the gate is using it13:39
*** jmontleon has joined #openstack-kolla13:41
sdakethere are two version in the repo rwellum13:41
sdakerwellum you can install kubernetes and kubelet 1.5.4 if you want to follow the instructions13:42
rwellumsdake: not blaming the guide -it's looking really good. In fact I thought it was my VM acting up.13:43
sdakesbezverk re kubeadm - did you try the version flag to install kubernetes-1.6.013:43
*** jaosorior has joined #openstack-kolla13:43
sdakerwellum not saying you did blame guide, only trying to help you get up and running :)13:43
rwellumsdake: maybe stupid question, why are we not using tools/setup_kubernetes.sh ?13:43
sdakesbezverk i suspect kubeadm will work better with teh proper version flag13:44
*** mbruzek has joined #openstack-kolla13:44
*** mbruzek has joined #openstack-kolla13:44
sdakerwellum we possibly could - I personally want to understand the workflow rather then some magic script13:44
*** mbruzek has quit IRC13:44
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Fixing 1.6.0 kubeadm init getting stuck  https://review.openstack.org/45139113:44
sdakerwellum setup_kubernetes.sh is designed to only run in the gate, not as a general purpose guide13:44
rwellumWell we have a few magic scripts - you wrote one to clean up k8s.13:44
rwellumOh IC.13:45
rwellumNow that would be a great comment at the top of the script...13:45
sdakerwellum ya that magic cleanup-k8s script is a little frightening :)13:45
sdakerwellum actually vhosakot wrote most of that13:45
sdakei debated putting it in the review at all13:46
sdakei may end up taking it out13:46
esharaoGM guys13:47
sdakesup esharao13:47
rwellumsdake: so in the guide we call out versions at the top - maybe we need to add those versions to the steps to install? We can always update the versions when tested.13:47
esharaowas retrying the guide.. with the new placement stuff called out by eanylin13:47
esharaostill my nova compute keeps on crashing but everything else is running..13:48
esharaomanaged to get my libvirt up once i removed profile from apparmour (maybe thats something to add for folks who use ubuntu)13:48
sdakeesharao i'm not sure eanylin tested it - he indicated he thought it was correct but was clear he wasn't sure last noight on irc13:49
*** jmontleon has quit IRC13:49
*** dciabrin has quit IRC13:49
openstackgerritMauricio Lima proposed openstack/kolla-ansible master: Pg_num should be configurable  https://review.openstack.org/45079713:49
esharaooh.. was worth a shot .. :) easy to helm delete and helm install..13:49
sdakerwellum ya - ther eis a flag to kubeadm to run a specific version13:49
*** jmontleon has joined #openstack-kolla13:49
sdakeesharao he did mention it had to do with the order in which they were launched13:49
sdakeesharao and I think thesame13:50
mgoddardsdake: thanks for the link, sorry for the slow reply. I'll comment on the that review - I'm not sure it's quite right.13:50
mgoddardsdake:  As for a backport, looks like it (or something similar) needs to go into stable/ocata but the issue should not be present in 4.0.0 thankfully.13:50
sdakemgoddard thanks - so do we need to revert thta change?13:50
sdakemgoddard you had indicated you had ironic working for binary for 4.0.013:50
*** rwallner has quit IRC13:51
sdakemgoddard or did you ahve it working for some other version of the containers?13:52
esharaosdake.. one question.. even though all other images have tag 4.0.0, my kolla-toolbox and haproxy have 3.0.2... Is that how its supposed to be?13:52
mgoddardsdake: I don't believe it warrants a revert - it should work. It just introduces an additional sudoers entry for the binary build that is unnecessary.13:52
sdakemgoddard trying to understand the impact  for 4.0.0 images specifically13:52
sdakeesharao haproxy should probably have a 4.0.0 tag13:52
sdakeesharao toolbox shoudl work fine with a 4.0.0 tag - although I haven't personally tested it13:53
mgoddardsdake: 4.0.0 should be fine - it did not include the iscsi_tcp modprobe that requires this sudoers entry13:53
sdakemgoddard cool so no backport needed13:54
mgoddardsdake: I haven't specifically tested ironic binary at 4.0.013:54
sdakeis this iscsp_tcp modprobe a new feature in pike?13:54
*** jrist has quit IRC13:55
sdakemgoddard ack - you tested 3.0.2 binary then?13:55
*** eanylin has joined #openstack-kolla13:55
mgoddardsdake: a backport is required because the iscsi_tcp modprobe was committed to stable/ocata after 4.0.013:55
sdakemgoddard thats a feature - you ha eto be kidding me!13:55
mgoddardsdake: IIRC I was testing ironic inspector13:55
sdakemgoddard fwiw, not lecturing you - jsut surprised there is a feature going into stable/ocata13:56
esharaosdake how do i force it since in my cloud.yaml tag is for 4.0.013:56
mgoddardsdake: it's a bug fix - iscsi-based ironic deployments fail without it13:56
sdakeesharao there is an order of precedent in cloud.yaml13:57
mgoddardsdake: although I wasn't involved in the fix13:57
sdakemgoddard ya - its not your fault - was just venting outloud sorry13:57
sdakeesharao the way the orderof precedent works is as follows13:57
mgoddardsdake: np :)13:57
sdakekolla.all.* is applied -> then kolla.service.* is applied13:57
*** jrist has joined #openstack-kolla13:57
sdakeesharao this precedence mens if you put the version in all, it will aply to all entries13:58
esharaoyes.. thats what i have13:58
sdakeif you want to override  the image tag for a sub-service, you have to do so13:58
*** srwilkers has joined #openstack-kolla13:58
sdakemorning srwilkers13:58
esharaohttp://paste.openstack.org/show/604662/13:58
srwilkersmorning sdake13:58
esharaooh i do not want to.. it just happened that way.. and was curious as to why it happened13:59
sdakeesharao i believe that should make all yoru images go to 4.0.013:59
sdakeesharao line 42 is the lowest precedence13:59
sdakeif you put an image_tag in one of the ot her services, it would take precedence over line 2414:00
sdake4214:00
eanylinesharao: I tested with 4.0.0 image14:00
sdakeesharao I should probably put that precedence explination in the docs14:00
eanylinbased on what I am seeing, cell0 should be executed after nova-compute is stable14:00
esharaosdake.. that would make sense..14:00
sdakeeanylin so a sync point is needed?14:01
esharaoeanylin.. i need to check as to how my tookbox took 3.0.214:01
sdakeesharao with that cloud.yaml you pasted, your toolbox is 3.0.2?14:01
eanylinsdake: I think so14:01
esharaoyes14:01
esharaolet me paste output of docker images14:01
sdakeok - well in that case the precedence ordering is not implemented correctly in the helm chart14:01
sdakelet me look at the code14:01
esharaohttp://paste.openstack.org/show/604663/14:02
eanylinI did another test, if I try and do nova-api-create-simple-cell-job before nova-compute it up14:02
eanylinit will crash14:02
*** dciabrin has joined #openstack-kolla14:02
eanylinwhich I think makes sense since it needs to know which hypervisor can be used14:02
*** Pavo has quit IRC14:04
*** zhurong_ has joined #openstack-kolla14:06
*** lucasxu has joined #openstack-kolla14:07
*** zhurong has quit IRC14:07
*** wirehead_ has quit IRC14:07
egonzalezeanylin yeah, simple cell first sync db with cell tables and then do a discover hosts which need compute nodes lookup by nova conductor14:08
*** wirehead_ has joined #openstack-kolla14:09
egonzalezmeeting in 1 or 2 hours? feeling lost after time change last week :/14:09
*** dciabrin has quit IRC14:09
sdakeegonzalez utc never changes14:10
sdakeegonzalez 2 hours14:10
eanylinegonzalez: Thanks14:10
sdakesbezverk quick q - why are these image tags specifiable: https://github.com/openstack/kolla-kubernetes/blob/master/helm/all_values.yaml#L11-L1414:10
egonzalezsdake thanks14:10
sdakesbezverk is it because they are in the common library?14:11
sdake12-14 that is14:11
sdakeegonzalez google recently added a way to determine times into their search tool14:13
sdakeegonzalez you can type in questions like "what time is 16:00UTC in location"14:13
sdakeand it should pop out the exact time14:14
sbezverksdake: because these are side car images, they belong to common section..14:14
sdakesbezverk right so side car images can't be overridden generally then?14:15
sdakesbezverk i think what would make sense is for image_tag to override sidecar image tags14:15
sbezverksdake: no14:15
sdakesbezverk that way people don't have to specify 4 different tags to override the defautls with one tag14:15
sdakesbezverk not asking you to do the work - curious if that is possible14:16
sdakesbezverk might be a good thing for me ot try out to learn how this stuff works ;)14:16
sbezverksdake: if you want to develop, then sure.. I do not see value in this..14:16
sdakesbezverk atm our image_tag default is 2.0.0 haproxy tagi s 3.0.2 and toolbox tag is 3.0.214:17
sdakesbezverk its kind of wonky to ask people to specify sidecar tag #'s since they may not even know what a sidecar container is :)14:17
sdaketag #S/tag ids14:17
*** zhurong_ has quit IRC14:18
sdakesbezverk is it simply a matter of modifying _common_defines.yaml?14:18
sbezverksdake: cannot say from the top of my head. I doubt it is a simple modification.. you will need to dig into it..14:19
*** rwallner has joined #openstack-kolla14:21
*** dciabrin has joined #openstack-kolla14:27
*** duonghq has joined #openstack-kolla14:27
duonghqevening guys14:27
*** shashank_t_ has joined #openstack-kolla14:28
*** magicboiz has joined #openstack-kolla14:34
*** adrian_otto has joined #openstack-kolla14:34
*** Pavo has joined #openstack-kolla14:38
*** krtaylor has joined #openstack-kolla14:39
*** fooliouno_ has quit IRC14:41
*** manheim has quit IRC14:44
rwellumsbezverk: so you think the kubelet 'issue' is a new feature? Won't be fixed.14:44
*** manheim has joined #openstack-kolla14:45
*** lamt has joined #openstack-kolla14:46
sbezverkrwellum: not sure whether it is a new feature or not, but it seems they change defaults or added it as a mandatory config..14:46
duonghqdoes our meeting time affect by Daylight saving time? due to its in UTC, so I think it doesn't?14:47
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla-ansible master: Update keystone haproxy config to balance based on source ip  https://review.openstack.org/45135814:49
*** matrohon_ has joined #openstack-kolla14:55
*** rwsu has quit IRC14:56
*** manheim has quit IRC14:56
sdakesbezverk ok - well it may  not have much value for kolla-kubernetes, but it would have value to me to learn how to write helm code14:58
sdakesbezverk so as long as folks don't block it - i'll do the work14:58
sdakesbezverk after I finish this deploy guide14:59
sdakeduonghq UTC does not take into account DST14:59
sdakeduonghq meeting is in 1 hour14:59
sdakesbezverk I think if you use       --use-kubernetes-version string         Choose a specific Kubernetes version for the control plane (default "stable")15:00
sdake you may get that value set automatically15:00
esharaosdake do you think its worth trying to change the toolbox to 4.0.0 in cloud yaml to see if it fixes my nova compute crashing15:00
sdakesbezverk i'd do this myself - however its school hour - can try in a bit15:00
duonghqsdake, roger15:00
*** Deys has quit IRC15:00
sdakeesharao you can change it but i believe that has zero to do with nova compute crashing15:00
esharaooh ok :( any other pointers15:01
sdakeesharao i'd question whether the toolbox 4.0.0 images work at all for kolla-kubernetes since nobody has tested that15:01
*** jistr is now known as jistr|mtg15:01
esharaoi think eanylin has 4.0.015:01
sdakeesharao the best we have so far is in the review, and eanylin seems to have sorted out the start order15:01
sdakeeanylin mind posting your cloud.yaml?15:01
sdakei've gotta get my kid rolling- bbiaf :)15:02
openstackgerritMerged openstack/kolla-ansible master: Improve developer documentation for Bifrost  https://review.openstack.org/43646215:02
*** zhubingbing has joined #openstack-kolla15:03
eanylinsdake: ok, actually it is similar to what we have, http://paste.openstack.org/show/604673/15:03
eanylinbut I need to mention that I have 2 versions of toolbox on my machine15:03
eanylin[root@kube03 ~]# docker images | grep toolbox15:04
eanylinkolla/centos-source-kolla-toolbox                        4.0.0               355042fb3f2b        2 weeks ago         733 MB15:04
eanylinkolla/centos-source-kolla-toolbox                        3.0.2               9f8d4f8130db        2 months ago        689 MB15:04
eanylin[root@kube03 ~]#15:04
*** matrohon_ has quit IRC15:05
*** matrohon_ has joined #openstack-kolla15:06
openstackgerritMerged openstack/kolla-ansible master: Revert "Fixes problem with kolla-kubernetes 3.0.2 images"  https://review.openstack.org/44435115:06
sdakeeanylin which oen is running?15:07
sbezverkgents, you do not need to worry about toolbox, as long as it is higher than 3.0.2, it is fine.. there was no changes done to this container recently15:07
sdakeeanylin it looks like with that config - toolbox 3.0.2 is running15:07
sdakeesharao ^^15:07
eanylinsdake: I may not have gotten the full context, what does that mean?15:07
eanylinsdake: Ok15:07
sdakesbezverk cool thanks for the confirmation - i remember something was done with toolbox - not sure which version it ultimately went into (if it went into master of kolla repos or not)15:08
*** Pavo has quit IRC15:08
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Try the using-kubernetes-version flag to fix gate  https://review.openstack.org/45145115:10
sbezverkgents, could you please run this command on your kube clusters "kubeadm version" and let me know if anybody has version lower than 207415:10
eanylinsdake: Seems like its using 3.0.2 toolbox15:10
eanylin[root@kube03 ~]# kubectl describe pod nova-compute-dxp6p -n kolla | grep Image15:10
eanylin    Image:docker.io/kolla/centos-source-kubernetes-entrypoint:4.0.015:10
eanylin    Image ID:docker-pullable://kolla/centos-source-kubernetes-entrypoint@sha256:e01591cfdb5def77ebc0f243b4d3fe7c94c19acac7ed41a73bdbca82fbcdd31215:10
eanylin    Image:docker.io/kolla/centos-source-kolla-toolbox:3.0.215:10
eanylin    Image ID:docker-pullable://kolla/centos-source-kolla-toolbox@sha256:fe10cce8325402a8c0a175ec88f11be13c8ff46430f6ef22a30c962c817adcbb15:10
eanylin    Image:docker.io/kolla/centos-source-nova-compute:4.0.015:10
eanylin    Image ID:docker-pullable://kolla/centos-source-nova-compute@sha256:7b6c58f8f7df7a44b732ddd9586a34fc1ece5c104c88ed3da247b4ba20737c4c15:10
eanylin    Image:docker.io/kolla/centos-source-nova-compute:4.0.015:10
eanylin    Image ID:docker-pullable://kolla/centos-source-nova-compute@sha256:7b6c58f8f7df7a44b732ddd9586a34fc1ece5c104c88ed3da247b4ba20737c4c15:10
eanylin[root@kube03 ~]#15:10
sdakeyup that makes sense15:10
eanylinyeah15:10
sdakeyour using toolbox 3.0.2 images15:10
*** Padouch has joined #openstack-kolla15:11
sdakeas toolbox is a sidecar and there is no way to globally override sidecar contianers atm15:11
eanylinsdake: My nova-compute been stable for past 2 hours15:11
duonghqJeffrey4l, zhubingbing do you have any idea about this error: http://logs.openstack.org/16/451016/2/check/gate-kolla-dsvm-deploy-ubuntu-source-ubuntu-xenial-nv/ced7a08/console.html#_2017-03-29_14_02_31_97048515:11
eanylin[root@kube03 ~]# kubectl get pods -n kolla | grep compute15:11
eanylinnova-compute-dxp6p                        1/1       Running   2          2h15:11
eanylin[root@kube03 ~]#15:11
sdaketoolbox 3.0.2 is not problematic - the biggest problem folks wiill run into atm is the fact that kubernetes 1.6.0 has been released and we are not sure if kubeadm is compatible15:11
duonghqI see it many times15:11
*** manheim has joined #openstack-kolla15:11
sdakeeanylin nice - I actually got compute running as well15:11
eanylinsdake: Agree15:11
sdakeeanylin what I wasn't able to do is create a vm - vm got stuck in BUILDING state15:11
*** dmsimard|afk is now known as dmsimard15:11
esharaohmm .. maybe i should just chuck my ubuntu and go with centos15:12
sdakeeanylin i got compute running by mostly running the instrucitons in the review15:12
eanylinyeah, the cluster did come up. I still had problems with VM creations though15:12
eanylinyeah, I did similar too15:12
*** sayantan_ has joined #openstack-kolla15:12
sdakeesharao it might make things simpler for the short term - although after we finish this guide adding ubuntu specific instructions might help :)15:12
eanylinsdake: Was trying to add comments along the way :)15:12
openstackgerritsean mooney proposed openstack/kolla master: introduces support for the OVS DPDK dataplane  https://review.openstack.org/34235415:12
sdakeeanylin yup eveyrone has been doing a great job on that review :)15:13
esharaook.. will re-start.. that will give a test to the doc as well15:13
rwellum+1 to using centos for now.15:13
esharaotalking about versions.. does the docker version matter?15:14
*** rwallner_ has joined #openstack-kolla15:14
esharaoi used the 17xx CE version15:14
esharaobut the guide says docker == 1.12.615:14
eanylinesharao: I am using 17xx CE version too15:15
Jeffrey4lduonghq, hrm, it is caused by a soft link. may be we should remove this feature.15:15
rwellumI'm a little confused with the cgroup change in kubectl if it's as per design (in which case in our document we either change input params, or point to a known earlier working release), or it's a bug that we just deal with. sbezverk suggests it's something we just deal with.15:15
duonghqJeffrey4l, ok15:15
duonghqJeffrey4l, softlink in logging?15:16
sdakeesharao indeed 17xx is incompatible with kubeadm15:16
sdakeesharao the preflight checks fail when you use 17xx15:16
sdakerwellum see: https://review.openstack.org/45145115:17
sdakerwellum i think for the guide we can just use kubernetes 1.5.515:17
Jeffrey4lduonghq, check this parameter http://docs.fluentd.org/v0.12/articles/out_file#symlinkpath15:17
sdakerwellum until the gate is sorted out15:17
sdakerwellum and iterate on the docs15:17
*** Padouch has quit IRC15:17
esharaook.. starting my centos testing now15:17
sdakerwellum i really believe that review above will not require any kind of workarounds as kubeadm is actively tested with kubernetes 1.6.015:18
*** rwallner has quit IRC15:18
duonghqJeffrey4l, thanks15:18
rwellumok sdake will try15:20
sbezverksdake: kubeadm is broken, at least version 270415:20
sbezverkrwellum: can you check your version of kubeadm? I need version lower than 270415:22
openstackgerritVikram Hosakote proposed openstack/kolla-kubernetes master: Deployment Guide Documentation  https://review.openstack.org/44735615:22
*** pbandark`brb has quit IRC15:24
rwellumhttps://www.irccloud.com/pastebin/9N36X8dO/15:24
sdakesbezverk upstream admit to this - or your speculation?15:24
rwellumsbezverk: ^^^15:24
*** blallau has quit IRC15:24
kfox1111morning.15:24
sdakesup kfox111115:24
sdakekfox1111 well you got your wish - kubernretes 1.6.0 is in the rpm repos ;)15:24
kfox1111sweet.15:24
sbezverksdake: many people are experiemncing the same issue15:24
kfox1111what broke? :)15:25
manheimif I am doing multinode deployment, is it absolutelly necesarry to do it with a registry server?15:25
sbezverkkfox1111: two things at least so far15:25
sdakekfox1111 kubeadm may be incompatible with 1.6.015:25
manheimcan't i just pull from docker hub?15:25
sdakemanheim you can pull from docker hub if yo ulike15:25
kfox1111sdake: no 1.6.0 kubadm rpm yet?15:25
sdakemanheim your performance will not be ideal15:25
sdakekfox1111 there is a 1.6.0 kubeadm rpm15:25
manheimseems fair :D15:25
sdakekfox1111 i've been using it for 3-4 days15:25
sbezverkkfox1111: there is and it breaks things15:25
kfox1111sdake: there's been an alpha for a long time.15:25
kfox1111sdake: should be a final release though.15:26
kfox1111k15:26
sdakesbezverk if you want to validate your assumption you can use yum to install a specific version15:26
sbezverksdake: it is not assuption, it a fact, I have the issue at the gate and on local cluster15:27
sbezverkkfox1111: can you run kubeadm version15:27
sdakei submitted a review for that sbezverk15:27
sdakehere is this repo data:15:27
sbezverkversion 2704 breaks init process15:27
sdake<version epoch="0" ver="1.6.0" rel="0"/>15:27
sdakethere is a 1.6.0 without version 270415:28
kfox1111sdake: ah. ok.15:28
sdakehttps://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64/repodata/filelists.xml15:28
kfox1111let me try and pull...15:28
sdake<package pkgid="5116fa4b73c700823cfc76b3cedff6622f2fbd0a3d2fa09bce6d93329771e291" name="kubeadm" arch="x86_64">15:28
sdake<version epoch="0" ver="1.6.0" rel="0"/>15:28
sdake<file>15:28
sdake</file>15:28
sdake<file>/usr/bin/kubeadm</file>15:28
sdake</package>15:28
sdake<package pkgid="082436e6e6cad1852864438b8f98ee6fa3b86b597554720b631876db39b8ef04" name="kubeadm" arch="x86_64">15:28
sdake<version epoch="0" ver="1.6.0" rel="0.alpha.0.2074.a092d8e0f95f52"/>15:28
sdake<file>15:28
sdake</file>15:28
sdake<file>/usr/bin/kubeadm</file>15:29
sdake</package>15:29
sdakethere are two versions in the repository15:29
sdake1.6.0-015:29
sdakeand 1.6.0-0.alpha.0.2074.*15:29
kfox1111-0? thats a bit odd.15:29
sdakeagreeed its wierd15:29
kfox1111maybe a testing rpm.15:29
*** fooliouno has joined #openstack-kolla15:29
sdakesup fooliouno15:29
fooliounosdake: morning sdake15:29
*** adrian_otto has quit IRC15:30
fooliounosdake: happy with the progress from yesterday. I see that you have newer versions of the guide15:30
sdake1.6.0-0.alpha may take priority over 1.6.0-015:31
sdakewith rpm15:31
sdakedo we know the alpha is being installed int eh gate?15:31
*** hrw has quit IRC15:31
sbezverkkfox1111: there is also new parameter which needs to be added to kubelet, see this ps  https://review.openstack.org/45139115:32
sdakekfox1111 https://review.openstack.org/#/c/451451/115:32
sdakesbezverk kubeadm should be doing that work15:32
sdakeor the kubelet packaging should be doing that work15:32
sdakesbezverk what you have there is a workaround for broken upstream packaging15:32
sbezverksdake: at least I have a workaround, what do you suggest?15:33
sdake2017-03-29 13:57:27.335915 | ---> Package kubeadm.x86_64 0:1.6.0-0.alpha.0.2074.a092d8e0f95f52 will be installed15:33
sdakesbezverk workaround is good15:33
sdakesbezverk helps fix the problem upstream in the upstream packaging if we can help them debug their mess15:34
sdakewhat I would suggest though is installing kubeadm-1.6.0-015:34
sdakelet me give that a go15:34
*** rwallner_ has quit IRC15:34
*** JoseMello has quit IRC15:35
openstackgerritMerged openstack/kolla master: make yum fail on missing packages  https://review.openstack.org/44908615:36
*** hrw has joined #openstack-kolla15:36
sbezverksdake: it is the final release of kubeadm is broken15:36
sdakesbezverk on my local machinie:15:37
sdake[sdake@kolla tools]$ sudo rpm -qa | grep kubeadm15:37
sdakekubeadm-1.6.0-0.alpha.0.2074.a092d8e0f95f52.x86_6415:37
*** matrohon_ has quit IRC15:37
sdakethe gate has exactly the same version15:37
sdakenote this version works for me locally15:37
sdakekbuernetes 1.6.0 upgrades probably break with kubeadm 1.6.0-0.alpha15:37
sdakei'd like to install just straight up kubeadm 1.6.0-015:38
*** klindgren__ has joined #openstack-kolla15:38
*** klindgren_ has quit IRC15:40
*** sayantan_ has quit IRC15:41
*** sayantan_ has joined #openstack-kolla15:41
*** rwallner_ has joined #openstack-kolla15:42
inc0good morning everyone15:43
spsuryamorning inc015:44
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Fixing 1.6.0 kubeadm init getting stuck  https://review.openstack.org/45139115:44
sbezverkinc0: good morning15:44
*** sayantan_ has quit IRC15:45
openstackgerritEduardo Gonzalez proposed openstack/kolla stable/newton: make yum fail on missing packages  https://review.openstack.org/45146515:46
duonghqmorning inc015:48
sdakekfox1111 sbezverk what i'd suggest is finding where that rpm package for kubeadm-1.6.0-0 is located on the internets15:48
sdakeI know its out there15:48
sdakei gott aget my kid to school bbl15:48
sbezverksdake: it turns out we need to specify version as 1.6.0-015:49
sbezverkthen yum takes non aloha version15:49
sbezverkoops non alpha15:49
sbezverkversion15:49
sbezverkI just pushed the fix to the gate to test15:50
*** sayantan_ has joined #openstack-kolla15:50
*** rmart04 has quit IRC15:55
*** targon has quit IRC15:56
sdakesbezverk pretty sure that is what I suggested :)15:56
*** manheim has quit IRC15:57
sdakethe reason you need to specify the versio nto yum is because -0.alpha takes precedence over -015:57
sdakewhoever did the versioning doesn't know how rpm works apprently :)15:57
sbezverksdake: there might be an issue with kubernetes hosted mirrors as despite specifying complete version, gate still installs alpha one16:00
*** vhosakot has joined #openstack-kolla16:00
kfox1111yeah. the alpha should be higher priority then -0 I think...16:00
sdakeright - when I do yum install kubernetes-1.6.0-0 it thinks the alpha version is good enough16:01
Jeffrey4lmeeting time, gusy.16:01
sdakethe way to typically work aroudn this is with a yum install http://uri16:01
sdakeI don't know what the uri is to the rpms16:01
sdakei looked in the repo location and don't immediately see the rpms16:02
*** adrian_otto has joined #openstack-kolla16:03
sbezverkman, what a PoS :( you resolve one issue to hit another..16:03
sbezverknow kubeadm does not label nodes correctly, so canal's etcd cannot start16:04
*** lamt has quit IRC16:10
*** rwallner_ has quit IRC16:12
*** dmsimard is now known as dmsimard|afk16:13
*** rwallner has joined #openstack-kolla16:13
kfox1111you can just psecify the version.16:16
kfox1111yum install kubeadm-1.6.0-016:16
*** rwallner has quit IRC16:17
kfox1111not seeing a kubeadm rpm yet...16:19
kfox1111where did you find 1.6.0?16:19
kfox1111oh... mayhbe its not printing it as its older.16:19
kfox1111on an already installed system, yum downgrade kubeadm-1.6.0-016:21
kfox1111kubeadm version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.0", GitCommit:"fff5156092b56e6bd60fff75aad4dc9de6b6ef37", GitTreeState:"clean", BuildDate:"2017-03-28T16:24:30Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}16:22
kfox1111yup. thats newer. :/16:22
sbezverkkfox1111: I used that exact command in the gate script16:22
sbezverkand got alpha still16:23
sbezverkkfox1111: I got it running on my cluster, by doing 2 changes16:23
esharaoanyone using 1.616:23
esharaoim running into issues when i do kubeadm -init16:24
sbezverk1 - kubelet config chance16:24
sbezverk2 - uninstalling alpha and installing normal kubeadm16:24
sbezverkesharao: please check this ps for the change you need to do BEFORE you run kubeadm init16:24
sbezverkhttps://review.openstack.org/45139116:25
kfox1111sbezverk: and the cgroup systemd thing?16:25
kfox1111or is that really needed? I would have thought it would default to systemd16:25
sbezverkkfox1111: it is in the PS16:25
sbezverkone extra parameters for kubelet16:25
kfox1111is it really needed though?16:25
sbezverkkfox1111: yep, otherwise kubelet fails to start16:26
sbezverkdue to misconfiugration16:26
sbezverkkfox1111: see this https://github.com/kubernetes/kubernetes/issues/43805#issuecomment-29012160716:27
kfox1111sbezverk: maybe specify it as 1.6.0-0.x86_6416:27
kfox1111maybe its matching the .alpha still.16:27
*** zhubingbing has quit IRC16:27
kfox1111sbezverk: ah. ok. thanks for the pointer.16:28
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Fixing 1.6.0 kubeadm init getting stuck  https://review.openstack.org/45139116:30
kfox1111sbezverk: next ps, could you rename it to be something like upgrade to 1.6? we probably should land a few other things in the ps too, like updating canal to the 1.6 version.16:32
sbezverkkfox1111: you bet, new canal.yaml uses different labeling and node selection, not compatible with old one16:34
esharaosbezverk - following the fix 451391 .. it says unknown flag: --apiserver-advertise-address16:34
kfox1111fun.16:34
*** egonzalez has quit IRC16:34
kfox1111esharao: its failing cause it wasn't successfully downgrading kubeadm to the newer version.16:34
kfox1111(parse that sentence carefully. :)16:34
mnaserJeffrey4l i fired up the builds16:34
sbezverkesharao: it is because you are still using alpha vgersion16:34
*** manheim has joined #openstack-kolla16:34
esharaolet me check..16:35
esharaooh thats right..16:35
mnaserbuilding ubuntu-binary right now, i'll do {ubuntu,centos,oraclelinux}-{binary,source}16:35
sbezverkkfox1111: yeah back to the future ;)16:35
kfox1111:)16:35
mnaserfor 3.0.3 first and then ill do mitaka after16:35
*** shardy has quit IRC16:35
Jeffrey4lmnaser, thanks a lots16:35
mnasernp16:35
*** duonghq has quit IRC16:36
sbezverkesharao: http://paste.openstack.org/show/604702/16:37
Jeffrey4lmnaser, what's your name in hub.docker.com?16:37
esharaogot it.. working.. i just uninstalled and installed16:38
Jeffrey4lusername or email.16:38
mnaserJeffrey4l mnaser16:38
hrwhttps://etherpad.openstack.org/p/hrw-multiarch-and-arm64-containers will have more data about my work16:38
mnaseremail is mnaser@vexxhost.com16:38
Jeffrey4lmnaser, added, you can push image to hub.docker.com/u/kolla now.16:38
*** manheim has quit IRC16:38
mnasercool, i'll show a paste of all the images once they've all been built locally and once i have an ok that things seem ok16:39
mnaserill push16:39
* hrw off16:39
Jeffrey4lgreat.16:40
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 uprgade  https://review.openstack.org/45139116:41
*** shashank_t_ has quit IRC16:41
*** rwallner has joined #openstack-kolla16:42
*** srwilkers has quit IRC16:44
esharaosbezverk: still see the same error - [apiclient] First node has registered, but is not ready yet16:45
*** unicell has quit IRC16:46
sbezverkesharao: kubeadm version16:46
*** sean-k-mooney has quit IRC16:47
*** shashank_t_ has joined #openstack-kolla16:47
*** jistr|mtg is now known as jistr16:47
esharaokubeadm version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.0", GitCommit:"fff5156092b56e6bd60fff75aad4dc9de6b6ef37", GitTreeState:"clean", BuildDate:"2017-03-28T16:24:30Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}16:49
*** shardy has joined #openstack-kolla16:52
*** matrohon_ has joined #openstack-kolla16:53
*** rhallisey has quit IRC16:58
*** rhallisey has joined #openstack-kolla17:00
*** srwilkers has joined #openstack-kolla17:00
sdakemnaser to build images17:01
sdakemnaser i typically run sudo pip install kolla==3.0.317:01
esharaoanyone else successful with kubeadm 1.6?17:01
sdakemnaser then build them17:01
sdakemnaser (in other words, I dont git checkout to build images - I use the pip distributed version of kolla)17:02
mnasersdake for "completeness" sake, i did pip install http://tarballs.o.o/kolla/kolla-3.0.3.tar.gz -- and building them now with different base and type17:02
sdakemnaser cool I dind't know that was a thing :)17:02
esharaobrb17:02
kfox1111esharao: focus is on getting it working in the gate right now I think.17:04
*** bmaltais has joined #openstack-kolla17:04
openstackgerritMerged openstack/kolla stable/newton: Install uwsgi in Centos binary  https://review.openstack.org/43568717:04
bmaltaisGood afternoon/morning17:05
*** matrohon_ has quit IRC17:05
bmaltaisI successfully installed openstack using kolla following the great quick start guide and I have a question17:05
bmaltaisI can't get the public network access to work17:06
bmaltaisI assigned the same subnet as my primary eth0 interface and assigned a range of available public IP17:06
bmaltaisbut any attempt to snat from a VM of associate a floating IP does not appear to work17:07
bmaltaisIs the private1 public network assocuate to the primary or the second NIC on the server?17:07
bmaltaisI know just enough about openstack to be dangerous I am afraid17:08
*** pbandark has joined #openstack-kolla17:08
sbezverkkfox1111: do you know ny any chance where ubuntu store this file? 10-kubeadm.conf17:09
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 uprgade  https://review.openstack.org/45139117:09
*** sayantan_ has quit IRC17:10
*** mrhillsman has quit IRC17:10
sbezverkgents, who run kube on ubuntu please let me know the location of this file 10-kubeadm.conf17:10
*** sambetts is now known as sambetts|afk17:11
*** mgoddard has quit IRC17:11
rwellumsbezverk: https://github.com/kubernetes/release/rpm/10-kubeadm.conf ?17:12
sbezverkrwellum: on your local cluster17:12
rwellumDarn I think I killed that VM - 1s...17:13
rwellumYeah sorry sbezverk17:14
*** rwsu has joined #openstack-kolla17:15
kfox1111sbezverk: not off hand.17:15
kfox1111only ubuntu k8s I've ever done is in the gate.17:15
sbezverkkfox1111: same here.. not sure where kubelet stores its startup file17:16
sbezverkon ubuntu17:17
sbezverkfor now the goal is to get centos at least in the gate working again17:17
kfox1111sbezverk: the gate script looks like it editied the same file on both distros.17:17
sbezverkkfox1111: nope it is multinode17:18
kfox1111so they use to be the same at lest.17:18
sbezverkand multinode is centos only17:18
*** vhosakot has quit IRC17:18
kfox1111sbezverk: thought you were talking about  10-kubeadm.conf17:18
kfox1111there's no conditional in the gate based on distro, so should be the same for all?17:18
sbezverkkfox1111: there is condition right at the top17:18
sbezverkkfox1111: I meant in setup_kubernetes.sh script17:19
kfox1111sbezverk: it ends at line 2717:19
openstackgerritMarcus Williams proposed openstack/kolla master: Add OpenDaylight Container to kolla  https://review.openstack.org/41636917:19
sbezverkkfox1111: it checkes if it is a master17:20
kfox1111that ends on line 6217:21
sbezverkplease look at the latest revision of my ps17:21
kfox1111k17:21
*** signed8b_ is now known as signed8bit_Zzz17:21
kfox1111sbezverk: looks ok to me.17:23
bmaltaisAny guide that show how to properly get a public network setup under kolla?17:23
sbezverkkfox1111: cool, let's see if this one will pass breaking point17:24
bmaltaisThe quickstart guide does not appear to implement a working public network17:24
kfox1111should get farther. :)17:24
kfox1111at least up until canal. :)17:25
bmaltaisTried modifying the init-runonce script but that did not work17:25
*** athomas has quit IRC17:25
*** sayantan_ has joined #openstack-kolla17:26
*** unicell has joined #openstack-kolla17:27
*** sayantan_ has quit IRC17:27
*** sayantan_ has joined #openstack-kolla17:27
sbezverkkfox1111: one step further but it stuck now of what other people reports, but I do not see it in my local cluster17:28
sbezverkkfox1111: [apiclient] First node has registered, but is not ready yet17:29
*** MasterOfBugs has joined #openstack-kolla17:29
rwellumbmaltais: not sure but i think that doc is out of date,  refer to https://review.openstack.org/#/c/447356/19/doc/source/deployment-guide.rst - which is also a WIP.17:29
bmaltaisrwellum: Thank's. I will read through it.17:32
openstackgerritMarcus Williams proposed openstack/kolla-ansible master: Split Openvswitch into own role  https://review.openstack.org/41679317:32
openstackgerritTim Clark proposed openstack/kolla-kubernetes master: WIP - Update nova-control service charts for Ocata  https://review.openstack.org/45149517:34
*** gfidente is now known as gfidente|afk17:36
*** tonanhngo has joined #openstack-kolla17:44
kfox1111sbezverk: looks like docker may need to be running before kubadm?17:45
*** rhallisey has quit IRC17:46
*** tonanhngo_ has joined #openstack-kolla17:48
*** tonanhngo has quit IRC17:48
*** rwallner has quit IRC17:48
sbezverkkfox1111: adding it..17:51
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 uprgade  https://review.openstack.org/45139117:51
*** pcaruana has quit IRC17:52
*** tonanhngo_ has quit IRC17:52
openstackgerritDave Walker proposed openstack/kolla master: Allow the use of previously built parent images  https://review.openstack.org/42632217:53
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 uprgade  https://review.openstack.org/45139117:53
*** tonanhngo has joined #openstack-kolla17:54
*** rhallisey has joined #openstack-kolla17:57
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: handle rtslib(-fb) package names and dependencies  https://review.openstack.org/45080518:00
sbezverkkfox1111: same thing on both ubuntu and centos, I added messages collection and syslog collection. hopefully when these jobs get killed, we can still access those logs to see WTF :(18:02
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 uprgade  https://review.openstack.org/45139118:04
*** mgoddard has joined #openstack-kolla18:05
*** lamt has joined #openstack-kolla18:09
*** lamt has quit IRC18:10
*** rwallner has joined #openstack-kolla18:11
*** lamt has joined #openstack-kolla18:12
*** mgoddard has quit IRC18:13
*** bmaltais has quit IRC18:22
rwellumsbezverk: kubeadm-1.6.0-0.x86_64 on centos did not get me past that cgroup issue18:23
sbezverkrwellum: even after you change kubelet start up config?18:24
rwellumsbezverk: must have missed a step then18:25
*** pbandark has quit IRC18:25
*** fooliouno has quit IRC18:27
sbezverkrwellum: I really hope so that you missed it otherwise we go back to square 118:27
*** rwallner has quit IRC18:28
sbezverkmeeting time, bb in 2 hours18:28
*** rwallner has joined #openstack-kolla18:30
openstackgerritOpenStack Proposal Bot proposed openstack/kolla-ansible master: Updated from global requirements  https://review.openstack.org/45101718:43
*** openstackstatus has joined #openstack-kolla18:44
*** ChanServ sets mode: +v openstackstatus18:44
*** lucasxu has quit IRC18:45
*** lucasxu has joined #openstack-kolla18:45
rwellumsbezverk: in case you see this during your meeting, with kubelet  correctly run your work-aroung worked for me.18:48
*** rwsu has quit IRC19:00
*** gfidente|afk has quit IRC19:03
*** common-- has joined #openstack-kolla19:07
esharaorwellum did you get your kube cluster up?19:08
rwellumesharao: giving it a go now.19:09
esharaook.. mine is not coming with 1.6 so let me know how it goes for you19:09
rwellumwith this being the magic kubelet invocation:19:09
rwellumhttps://www.irccloud.com/pastebin/7UtR4uMb/19:09
*** rwallner has quit IRC19:11
common--Trying to follow https://docs.openstack.org/developer/kolla/newton/quickstart.html -- it does not appear to say to install kolla-ansible, so if you go through it, it will fail on kolla-genpwd19:12
rwellumesharao: which part do you fail on?19:13
esharaokubeadm init keeps on waiting.. - [apiclient] Created API client, waiting for the control plane to become ready19:14
rwellumsudo tail /var/log/messages ?19:16
rwellumhttps://www.irccloud.com/pastebin/MH1lrNN8/19:16
rwellumMine is stuck there too19:16
esharaokubelet cgroup driver: "systemd" is different from docker cgroup driver: "cgroupfs"19:16
rwellumAh did you see the snippet above - with sbezverk fix for this?19:16
rwellumhttps://www.irccloud.com/pastebin/7UtR4uMb/19:17
*** signed8bit_Zzz is now known as signed8b_19:17
esharaoone min19:17
esharaorechecked my config..19:18
esharaothe file has sbezverk's fix19:19
rwellumHmmm I got past that, but stuck later on.19:19
rwellumwith that fix.19:19
*** david-lyle has quit IRC19:19
*** david-lyle has joined #openstack-kolla19:19
rwellumYou also used kubeadm-1.6.0-0.x86_64 ?19:19
esharaoyes.. initially it went to alpha and then i uninstalled an specifically installed the 1.6.019:21
rwellumDocker version? I have: 1.12.619:22
rwellumNot sure if that's important tho19:22
esharaooh i have 17.03.1-ce19:24
EmilienMsdake, inc0: can you make sure we have some Kolla folks at the session from [deployment][forum] proposing a session about future of configuration management - ops + devs wanted!19:24
rwellumWould be the next step I think esharao , as the dep guide still says 1.12.6 - but I'm unsure because the 17.x is dockers new release model. Hopefully sdake or sbezverk can help here.19:26
sdakeEmilienM inc0 said he would handle that forum request19:26
sdakeEmilienM i think in terms of kolla folks - people go wher ethey want during summit - it was always challenging for me to get poeple to even come to htte kolla sessions that were kolla cores as ptl :)19:26
esharaohmm rwellum i believe eanylin got it up based on 17.x19:26
esharaobut with kubeadm 1.5.519:26
EmilienMsdake: ack19:27
*** david-lyle has quit IRC19:27
*** shashank_t_ has quit IRC19:28
*** david-lyle has joined #openstack-kolla19:30
openstackgerritBertrand Lallau proposed openstack/kolla-ansible master: Trove fix backup restore with Swift  https://review.openstack.org/45153819:31
*** matrohon_ has joined #openstack-kolla19:32
*** openstackgerrit has quit IRC19:33
*** openstackgerrit has joined #openstack-kolla19:33
openstackgerritBertrand Lallau proposed openstack/kolla-ansible master: Trove fix backup restore with Swift  https://review.openstack.org/45153819:33
rwellumI'm willing to try that esharao but not sure how to get that version of docker yet.19:34
esharaohttps://docs.docker.com/engine/installation/linux/centos/ - and installed docker ce19:34
esharaoif that is what you were asking19:35
sdakerwellum esharao re docker 17.xx - when I tried kubeadm 16.0.alpha - it failed the preflight checks with docker 170019:35
sdakeso I had to install distro provided docker 12.619:36
sdakei have no idea what the behavior of kubeadm1 16.0 release is19:36
sdakeas I have not tried it yet19:36
esharaosdake i used the https://review.openstack.org/#/c/45139119:36
sdakerather 1.16.0-alpha and 1.16.0  release19:36
esharaoand made change to 1.6.0-0 release by installing it with release name19:37
sdakei dont think it is wise to skip preflight checks esharao19:38
esharaokubeadm-1.6.0-0.x86_6419:38
sdakepreflight checks check compatibility and other various things19:38
sdakeif they complain docker 17 is not supported, likely a reason for that19:38
esharaohmm sdake i tried it based on what sbezverk told19:38
sdakei couldn't get 13.0 to work at all even though kubeadm worked19:38
esharaohmm ok.. let me restart again with docker 1.12.6 and kubeadm 1.5.x and see how it goes19:39
*** rwallner has joined #openstack-kolla19:39
openstackgerritBertrand Lallau proposed openstack/kolla-ansible master: Set to Unix format trove-conductor.conf file  https://review.openstack.org/45153919:40
*** lamt has quit IRC19:41
*** mcnanci has quit IRC19:45
sdakekubeadm 1.6.0 may work with docker 1.12.6 or docker 17xx - just dont know19:53
sdakei used kubeadm 1.6.0-alpha to produce those docs19:53
inc0sdake: there are few forum requests19:53
*** matrohon_ has quit IRC19:54
inc0EmilienM: I'll be there for sure, thanks for heads up19:54
EmilienMcool19:54
sdakeinc0 you said you would handle it - wfm - i was simply pointing out the call for forum requests was clsoing :)19:56
*** Serlex has quit IRC19:56
rwellumesharao: how do you get kubeadm 1.5.x?19:57
sdakeguys - i would recommend kubeadm 1.6.0-release19:58
esharaodid not.. that was yesterday.. :(19:58
*** lamt has joined #openstack-kolla19:58
esharaoi used 1.6 and docker 1.12.619:58
sdakei would also recommend kubernetes 1.5.4 for now19:58
esharaobut am running into an issue19:58
esharao[apiclient] First node has registered, but is not ready yet19:58
sdakeuntil people sort out the issues with kubernetes 1.619:58
rwellumYeah that's my issue esharao19:58
esharao:(19:59
esharaoi see in my /var/log - Unable to update cni config20:00
*** manheim has joined #openstack-kolla20:00
rwellumYeah that's it.20:01
rwellumI don't know the magic version combinations - I think I've tried 5 different combos..20:01
*** rwallner has quit IRC20:02
*** fooliouno has joined #openstack-kolla20:04
sbezverkrwellum: thanks a lot Rich for confirming it..20:06
rwellumsbezverk: but then I (and esharao ) hit another issue it seems.20:06
rwellumhttps://www.irccloud.com/pastebin/MH1lrNN8/20:07
*** pron has quit IRC20:07
srwilkersyeah, that issues a pain in my ass20:07
sbezverkrwellum: same at the gate :(20:08
esharaook.. so maybe i go back to kub 1.5.4 and try it again20:08
*** pron has joined #openstack-kolla20:09
sbezverkrwellum: by some reason kubelet is expecting to have networking up even before it got deployed..20:09
srwilkersseems to be related to the changes made to the kubelet and how it's determining when a pod's ready20:09
srwilkerssbezverk, yep. nailed it20:09
kfox1111back for a few.20:09
rwellumYeah looks like it - nasty20:09
srwilkerswith the changes, the master node never registers that its ready20:09
*** rwallner has joined #openstack-kolla20:09
sbezverkrwellum: there is an issue opened for the same reason with kube folks20:09
sbezverksrwilkers: the funny thing is, it worked one on my local test bed20:10
sbezverkI cannot see what I have done differently though20:10
kfox1111the non cni plugin is still there. could it just be turned off for now?20:10
kfox1111well, are we talking network plugin, or container engine plugin?20:11
sbezverkhttp://paste.openstack.org/show/604730/20:11
kfox1111sbezverk: nice. :)20:12
sbezverkkfox1111: nope, because I have no idea why it worked on my cluster20:12
srwilkerssbezverk, ive been fighting with it the better part of half a day with no luck20:13
srwilkersseems theres urgent interest on the issue in github to get it fixed, which is nice20:13
sbezverksrwilkers: yeah lots of folks reported the same issue20:13
rwellum A race condition maybe? Might explain why it works occasionally.20:14
esharaorwellum did you figure out how to install the previous version of kubeadm since 1.6 is broken20:20
*** krtaylor has quit IRC20:20
sdakeesharao you can yum install the specific version20:21
rwellumNo I was hoping you would. :)20:21
esharaooh i tried it..20:21
esharaoit did not work sdake20:21
sdakeesharao didn't work as in didn't install or kubeadm didn't come up?20:21
srwilkersthe problem is that the previous version of kubeadm still requires kubelet 1.620:21
kfox1111sbezverk: bummer.20:21
esharaodid not install (did not find)20:21
srwilkerswhich is whats causing the breaj20:21
srwilkersbreak, even20:21
kfox1111sbezverk: ok, so.... does 1.5 still work?20:22
kfox1111just wondering if the gate's broken.20:22
kfox1111if 1.6 doesn't work for a few days, but the gate still works, not the end of the world.20:22
kfox1111if the gate's broken, a whole nother thing.20:22
sbezverkkfox1111: we would need to pin 1.5.6 binaries then20:25
sdakeif there is alot of interst upstream in kubernetes in fixing it20:26
sbezverksrwilkers: since you have the issue locally, could you try to remove KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"20:26
sdakewe shoud let them fix it - but provide what we know thus far20:26
sbezverkfrom kubelet command like20:26
sbezverkline..20:26
sdakewe can pin for a f ew days until a new release is cut or the docs are updated20:26
rwellum"previous version of kubeadm still requires kubelet 1.6" - that's an issue too right?20:27
sdakerwellum here are my deps:20:28
*** Guest6666 has quit IRC20:28
*** Guest6666 has joined #openstack-kolla20:29
srwilkersrwellum, yes. thats an issue20:29
kfox1111sdake: yeah. it could easily take a few days to iron out by the current state of things.20:29
kfox1111if thats the case, we should fix any breakage now by pinning back.20:29
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 uprgade  https://review.openstack.org/45139120:29
kfox1111if we think we can fix 1.6 in the next couple hours though, we should probably just do that.20:30
*** rwallner has quit IRC20:30
sbezverkkfox1111: let's try to play with networking options20:30
sbezverkwho has locally reproduced the issue and willing to try a few things?20:30
rwellumI can do it sbezverk - just a VM I can revert when it all goes belly-up.20:31
kfox1111I'm fine pressing forward to 1.6 either way. just wondering when we shoudl cut our losses and just submit a gate fix for 1.5.20:31
*** pramodrj07 has joined #openstack-kolla20:31
esharaowhat do you want to try sbezverk20:32
sbezverkkfox1111: let's try to workaround it until the eob20:32
kfox1111sbezverk: kk. sounds good.20:32
sdakerwellum having trouble logging into my local network - when i can I'll list out my deps in a few minutes20:32
*** david-lyle has quit IRC20:33
sdakesbezverk so your planning to get gate working with kubernetes 1.6 then, or pin to 1.5.5?20:33
sbezverkesharao: rwellum: find kubelet startup file and make sure this env variable KUBELET_NETWORK_ARGS is romoved20:33
rwellumsdake: no worries, but we are in addition having issues installing older versions - if you have the secret sauce for that it would help.20:33
sbezverksdake: yes20:33
sdakerwellum indeed I do have the secret sauce20:33
*** MasterOfBugs has quit IRC20:34
sdakerwellum its not secret - just have to know how to use rpm :)20:34
sdakerwellum keep in mind by use - i mean be expert in - i worked at rht for 9 long long years20:34
sdakethats a 64bit there ;)20:34
rwellumcool sdake20:35
*** rwallner has joined #openstack-kolla20:35
*** david-lyle has joined #openstack-kolla20:35
rwellummost people I know went the other direction...20:35
rwellumvoluntarily and not so.20:35
sdakerwellum which other direction20:35
sdakeyou mean deb?20:36
rwellumfrom your place of work to rh20:36
sdakenot surprising - people change employers in our indsutry from one to another20:36
sdakeand even back again :)20:36
sdakerht is a great place to work - i enjoyed it20:37
sdakecisco is also a great place to work - i enjoy it20:37
sdakenothing really bad to say about either co20:37
rwellumlucky either way20:38
sbezverkgents, are you trying to reconfigure kubelet or enjoying sdake speach ;)20:38
rwellumsbezverk: I am trying to find KUBELET_NETWORK_ARGS20:38
srwilkers;)20:39
sdakeI'll tell you i don't regret getting off the linux desktop :)20:39
sdakehttp://paste.openstack.org/show/60473620:39
srwilkersthats not secret sauce, thats running the old version of kubelet that isnt broken20:40
*** david-lyle has quit IRC20:40
sdakesrwilkers yes - I didn't say it was secret sauce20:40
esharaosbezverk it worked20:41
esharaobut kubectl get pods is not connecting20:41
*** david-lyle has joined #openstack-kolla20:42
sbezverkesharao: one sec what worked?20:42
esharaomy kubeadm init worked20:42
sbezverkwhen you do kubectl get pods what do you get?20:43
esharaohttp://paste.openstack.org/show/604737/20:43
esharaothats the problem - kubectl says cannot connect to 808020:43
sbezverkesharao: that is notmal20:43
esharaoThe connection to the server localhost:8080 was refused - did you specify the right host or port?20:43
sbezverkyou need to change manifest20:43
esharaooh is it inyour patch?20:44
sbezverkvi /etc/kubernetes/manifest/kube-apiserver.yaml20:44
*** cu5 has joined #openstack-kolla20:44
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Pin gate to working versions of kubernetes 1.5.4 and kubeadm  https://review.openstack.org/45155620:44
sdakerwallner that command should work - might wait for the gate to do its thing20:45
sdakerwallner i really dont want to uninstall/reinstall to test it as i have working functional copies ;)20:45
sbezverkadd this line220:45
sbezverk    - --insecure-port=808020:45
sbezverk    - --insecure-bind-address=0.0.0.020:45
sdakerwellum although the gate should return successful results for centos if that works ^^20:45
sbezverkafter line - --kubelet-client-key=20:46
*** david-lyle has quit IRC20:46
esharaodone20:47
*** rwallner has quit IRC20:47
sbezverkesharao: when you exit vi, apiserver should automatically gets restarted20:48
sbezverkjust give it a couple of minutes20:48
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Pin gate to working versions of kubernetes 1.5.4 and kubeadm  https://review.openstack.org/45155620:48
sbezverkand watch for port 8080 listening on 0.0.0.020:48
esharaohttp://paste.openstack.org/show/604738/20:49
esharaoits working now20:49
sbezverk:-)20:49
sbezverknow we need to see if we can get networking running again20:49
esharao:)20:50
*** rwsu has joined #openstack-kolla20:50
esharaowant me to try the canal stuff?20:50
sbezverkesharao: you need to wget this file https://raw.githubusercontent.com/projectcalico/canal/master/k8s-install/kubeadm/1.6/canal.yaml20:51
sbezverkesharao: then change whatever ip info you want in there aand then we will try to create canal thingy..20:51
esharaoyep doing that20:52
*** david-lyle has joined #openstack-kolla20:52
*** rwallner has joined #openstack-kolla20:53
rwellumsdake: do we need to add that patch to the dep doc?20:53
*** srwilkers has quit IRC20:53
esharaosbezverk - all running20:54
sbezverkyou mean all or all all all ;)?20:54
esharaohttp://paste.openstack.org/show/604740/20:55
sbezverkesharao: that is a beautyyyyyyyy20:55
esharao:D20:55
*** cu5 has quit IRC20:55
esharaoso continue with kolla?20:56
esharaofollowing sdake DG20:56
*** rwallner has quit IRC20:57
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 uprgade  https://review.openstack.org/45139121:01
sbezverklet's see if this recepie helps in the gate21:01
esharaosbezverk - any idea on taint nodes failing?21:02
sbezverkesharao: you need to check kube 1.6.0 change log, I think I saw something there about taints, do not remember what exactly21:03
esharaook..21:04
*** cu5 has joined #openstack-kolla21:04
hawiquestion: is there any chance to debug why mariadb doesnt come up properly after "Waiting for MariaDB service to be ready"? in somehow one moment all mariadb instances were not listening on 3306 port at all. i have no idea how it happened, but i really dont want to reinstall it, i want to understand, what's wrong with it21:06
*** rwsu has quit IRC21:06
*** mgoddard has joined #openstack-kolla21:06
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 uprgade  https://review.openstack.org/45139121:07
*** rhallisey has quit IRC21:08
SamYaplehawi: if the cluster died then it can't recover on its own under most situations21:09
*** jrobinson has joined #openstack-kolla21:09
SamYaplehawi: it could have been network disruption on your end that caused the nodes to lose quorum, but if no node is listening the clsuter is down right now21:09
SamYapleyoull have to run the recovery playbook21:09
hawiSamYaple: i tried, nothing happened21:09
sbezverkesharao: kubectl taint nodes kolla.sbezverk.cisco.com node-role.kubernetes.io/master:NoSchedule-21:09
sbezverkit is new in 1.6.021:10
SamYaplehawi: the recovery playbook expects all running mariadb docker containers to be stopped21:10
esharaoahh ok..21:10
hawiSamYaple: you mean mariadb_recovery yes?21:10
esharaono schedule..21:10
SamYaplehawi: yes21:10
rwellumsdake: pinning like you suggested worked for me21:10
rwellumhttp://paste.openstack.org/show/604742/21:10
esharaothat did the trick sbezverk21:11
hawiSamYaple: i'm really impressed! in somehow it didnt work for several hours back21:13
*** mgoddard has quit IRC21:14
hawibut now ... it's good...21:14
SamYaplehawi: i hacked your system and fixed it while you werent looking21:14
*** krtaylor has joined #openstack-kolla21:14
hawi:)21:14
hawidunno...perhaps i didnt stop containers. i got compains about my.cnf and galera.j2 differ, also about  gcs connect failed: Connection timed out21:16
SamYaplestopping the containers is a must if its broken like you described21:17
*** rwsu has joined #openstack-kolla21:20
*** jtriley has quit IRC21:24
*** yee379 has joined #openstack-kolla21:25
*** lp|Kubecon has quit IRC21:25
*** kolla-slack has quit IRC21:25
*** kolla-slack1 has joined #openstack-kolla21:25
*** yee37915 has quit IRC21:26
*** eanylin has quit IRC21:26
*** eanylin has joined #openstack-kolla21:26
*** mattmceuen has quit IRC21:28
*** lazyPower has joined #openstack-kolla21:28
*** lucasxu has quit IRC21:34
hawiSamYaple: thanks. didnt expect so simple solution, as i already spend couple of hours on trying to figure out what's wrong21:34
fooliounoI tried to create a VM, but got an error: "No valid host was found. There are not enough hosts available"21:36
fooliounoHowever, there is a node listed under hypervisors21:37
eanylinI just got my Cirros VM up :)21:37
fooliounoenaylin: any changes needed?21:37
fooliounoeanylin: congrats :)21:37
eanylinmy kubeadm version is v1.6.0-alpha.0.2074+a092d8e0f95f5221:37
inc0fooliouno: check nova scheduler log21:37
eanylindocker version is v1.6.0-alpha.0.2074+a092d8e0f95f5221:37
eanylin17.03.0-ce21:38
eanylinsorry21:38
fooliounoinc0: Let me check that21:38
eanylindocker version is 17.03.0-ce21:38
sdakeeanylin grats ddue :)21:38
sdakeeanylin replicated the gate outside the gate ;)21:38
sdakeeanylin mind adding any comments to the review - and i'll sort em out tonight?21:39
eanylinsdake: I had to do what is mentioned here https://bugs.launchpad.net/packstack/+bug/166126721:39
openstackLaunchpad bug 1661267 in Packstack "libvirtd crashes while launching an instance from openstack horizon" [High,Fix released]21:39
sdakeso we can get this documentation merged21:39
eanylinI believe we may still hit problems due to https://bugs.launchpad.net/packstack/+bug/166126721:39
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 uprgade  https://review.openstack.org/45139121:39
eanylinsdake: sure21:40
sdakecpu_mode=none21:40
* sdake GROANS21:40
eanylinyes21:40
eanylinit broke my hypervisor connection21:40
eanylinbecause of that21:40
eanylinI dont know if it will work properly if its on bare metal, didnt have one to test21:41
eanylinI am using Centos7 VM and had this problem21:41
*** pramodrj07 has quit IRC21:41
*** pramodrj07 has joined #openstack-kolla21:41
eanylinhad to manually update nova.conf in nova-compute to get around that21:41
*** mcnanci has joined #openstack-kolla21:42
*** schwicht has quit IRC21:43
*** ruhe has quit IRC21:44
*** timss has quit IRC21:44
*** timss has joined #openstack-kolla21:45
fooliounoinc0: nova-scheduler says that the filter RetryFilter returned 0 hosts21:48
inc0fooliouno: hmm...from top of my head I can't remember reason for that21:48
inc0check nova-compute log21:48
fooliounook ..21:48
inc0I have one candidate in mind21:49
*** ruhe has joined #openstack-kolla21:49
eanylinfooliouno: Did you set up the cell0 already?21:50
*** shardy has quit IRC21:50
fooliounoenaylin: the cell0-create-db pod completed21:51
eanylinhow about nova-api-create-simple-cell-job ?21:53
*** erlon has quit IRC21:55
*** sayantan_ has quit IRC21:55
fooliounoeanylin: I see a nova-api-create-cell (not simple-cell) and that job completed successfully21:55
eanylinok21:56
fooliounoeanylin: I need to head home, but will login later in the evening.21:56
eanylinfooliouno: Sure21:57
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 uprgade  https://review.openstack.org/45139121:58
*** fooliouno has quit IRC21:59
*** esharao has quit IRC22:00
*** sayantan_ has joined #openstack-kolla22:04
*** schwicht has joined #openstack-kolla22:09
*** adrian_otto has quit IRC22:09
*** adrian_otto has joined #openstack-kolla22:10
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 uprgade  https://review.openstack.org/45139122:18
*** sayantan_ has quit IRC22:21
*** sayantan_ has joined #openstack-kolla22:21
*** rwsu has quit IRC22:23
*** manheim has quit IRC22:32
*** shashank_t_ has joined #openstack-kolla22:33
*** rwsu has joined #openstack-kolla22:35
*** sayantan_ has quit IRC22:36
*** krtaylor has quit IRC22:38
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 uprgade  https://review.openstack.org/45139122:38
sbezverkkfox1111: almost there :) last piece, fixing tainting..22:41
*** schwicht has quit IRC22:47
*** shashank_t_ has quit IRC22:54
*** shashank_t_ has joined #openstack-kolla22:55
*** sayantan_ has joined #openstack-kolla22:58
kfox1111back.22:58
kfox1111sbezverk: nice. :)22:58
*** shashank_t_ has quit IRC22:59
sbezverkkfox1111: they also changed token format, man it is crazy..23:00
kfox1111sbezverk: https://raw.githubusercontent.com/projectcalico/canal/master/k8s-install/kubeadm/1.6/canal.yaml ?23:00
kfox1111heh. yeah. I heard that.23:00
kfox1111one of the drawbacks on using something thats alpha.23:00
kfox1111(now beta)23:00
sbezverkkfox1111: http://logs.openstack.org/91/451391/17/check/gate-kolla-kubernetes-deploy-centos-binary-3-ceph-multi-nv/5ddb440/console.html#_2017-03-29_22_48_46_10994523:00
kfox1111sbezverk: did we fail to parse out the token?23:02
sbezverkyep23:02
sbezverkbut they give reg exp23:02
sbezverk"^([a-z0-9]{6})\\.([a-z0-9]{16})$"23:02
sbezverkI will use it instead23:02
sbezverkof what we have now23:02
kfox1111hmm....23:02
kfox1111no, I think that will break...23:02
kfox1111give me a sec...23:02
sbezverkok23:03
kfox1111so, we have: kubeadm join --token 6aba5e.876011764b2ae7b7 10.32.180.247:644323:03
kfox1111oh...23:04
kfox1111they removed '=' from the string.23:04
kfox1111used to be --token=....23:04
kfox1111now its --token ....23:04
kfox1111that makes it easier to parse. :)23:04
kfox1111so, on line 62, drop out: | sed 's/[^=]*=//'23:05
kfox1111change $3 to $4,23:05
kfox1111and $4 to $5.23:05
kfox1111should fix it.23:06
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 uprgade  https://review.openstack.org/45139123:07
sbezverkkfox1111: ok let me try it23:08
sbezverkkfox1111: but the format is blah.blahblah now23:09
sbezverkah ok it will grab whole token23:10
kfox1111right. :)23:11
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 uprgade  https://review.openstack.org/45139123:13
*** pramodrj07 has quit IRC23:22
kfox11112017-03-29 23:22:55.348903 | Error from server (Forbidden): User "system:node:centos-7-osic-cloud1-s3700-8170757" cannot list jobs.batch in the namespace "kube-system". (get jobs.batch)23:25
kfox1111fun...23:25
kfox1111sbezverk: might want to disable rbac for now.23:26
sbezverkkfox1111: man, will it eve stop ?!?!23:27
kfox1111hehe. a lot of new things in 1.6.23:27
kfox1111but so far, less then a day to fix. not too bad really.23:27
sbezverkkfox1111: well we do not know what will hit us next23:29
kfox1111thats why we were talking about putting a pin back to 1.5 just in case this is too deep a rabbit hole.23:30
kfox1111https://github.com/tailnode/kubernetes.github.io/commit/1c04cedd412b31bbcd8b708e8ac1033fac8e31a623:30
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 uprgade  https://review.openstack.org/45139123:31
kfox1111looks like you can set authorization mode but not sure you can set it via the commandline.23:31
kfox1111will that work?23:31
sbezverkkfox1111: yeah if by the end of day is not working, then go back to 1.523:32
*** jtriley has joined #openstack-kolla23:32
kfox1111does rbac need to be replaced with the allow plugin? or something different?23:32
kfox1111do you have a copy of kube-apiserver.yaml somewhere?23:33
sbezverkkfox1111: I removed rbac and I think it should fall back23:33
sbezverkkfox1111: sure23:33
kfox1111it may need to be --authorization-mode=AlwaysAllow23:33
sbezverkhttp://paste.openstack.org/show/604756/23:35
kfox1111cool. thx.23:35
kfox1111yeah. I'm guessing the sed should be /RBAC/AlwaysAllow/  but I guess we'll see. :)23:36
kfox1111I'm a little surprised its using system:node:centos-7-osic-cloud1-s3700-8170757 though...23:37
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 uprgade  https://review.openstack.org/45139123:38
kfox1111I'd expect it to be the admin by default...23:38
kfox1111kind of weird.23:38
sbezverkkfox1111: strange on my cluster I can run command even with rbac23:39
kfox1111yeah, kind of weirded helm would be tryign to use the node creds.23:39
*** lamt has quit IRC23:39
kfox1111I'm guessing we will hit a bunch of other problems with rbac enabled though.23:39
kfox1111so turning it off for now is agood thing.23:40
sbezverkbut I changed to AlwaysAllow23:40
kfox1111should have a whole different ps for figuring out how to enable it.23:40
kfox1111maybe a set of ps's.23:40
*** schwicht has joined #openstack-kolla23:40
sdakei've got a patch in the queue for 1.523:46
sdakedon't have ubuntu tho - so i guess i'll sort that out in some way23:47
*** hieulq has quit IRC23:47
sdakehttps://review.openstack.org/#/c/451556/23:47
sbezverkkfox1111: it went further :)23:49
kfox1111you sure? one of them failed.23:50
kfox1111centos-binary-2-ceph-nv failed.... maybe random...23:50
kfox1111which one you looking at?23:51
*** adrian_otto has quit IRC23:51
kfox1111ovs failed too...23:51
kfox1111hmm.. thought that one did get farther...23:51
sbezverkkfox1111: we should use port :8080 for inside communication23:52
kfox1111not sure I follow.23:53
kfox1111http://logs.openstack.org/91/451391/21/check/gate-kolla-kubernetes-deploy-centos-binary-2-external-ovs-nv/b16b427/logs/pods.yaml is very empty...23:53
kfox1111weird.23:53
sbezverkkfox1111: because it tries to connect to 644323:54
kfox1111which thingy?23:59
kfox1111kubectl should use whatever kubeadm configured by default?23:59
kfox1111it always has in the past at least.23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!