Monday, 2019-03-11

*** ricolin has joined #openstack-containers00:59
*** hongbin has joined #openstack-containers02:02
*** ykarel has joined #openstack-containers03:22
*** hongbin has quit IRC03:58
*** udesale has joined #openstack-containers04:03
*** ramishra has joined #openstack-containers04:45
*** janki has joined #openstack-containers04:46
openstackgerritGhanshyam Mann proposed openstack/magnum master: DNM: Testing all legacy jobs on bionic  https://review.openstack.org/64233905:15
*** jaewook_oh has joined #openstack-containers05:15
*** jaewook_oh has quit IRC05:22
*** sapd1 has joined #openstack-containers06:08
*** ivve has joined #openstack-containers06:28
*** ianychoi has quit IRC06:32
*** ianychoi has joined #openstack-containers06:32
*** ianychoi has quit IRC06:35
*** ianychoi has joined #openstack-containers06:36
*** pcaruana has joined #openstack-containers07:00
*** rcernin has quit IRC07:03
*** belmoreira has joined #openstack-containers07:41
*** belmoreira has quit IRC07:42
*** belmoreira has joined #openstack-containers07:42
*** flwang1 has joined #openstack-containers07:55
flwang1strigazi: around for a catch up?07:56
openstackgerritFeilong Wang proposed openstack/magnum master: (WIP) Add cluster upgrade to the API  https://review.openstack.org/51495907:56
*** gsimondon has joined #openstack-containers08:10
*** sapd1 has quit IRC08:15
*** gsimondo1 has joined #openstack-containers08:29
*** gsimondon has quit IRC08:31
openstackgerritMerged openstack/python-magnumclient master: Add hidden property to cluster template  https://review.openstack.org/63495508:47
*** adrianreza has quit IRC09:00
*** adrianreza has joined #openstack-containers09:00
*** ykarel has quit IRC09:10
*** ykarel has joined #openstack-containers09:11
*** ttsiouts has joined #openstack-containers09:26
*** janki has quit IRC09:30
*** jchhatbar has joined #openstack-containers09:30
*** ykarel is now known as ykarel|lunch09:37
*** alisanhaji has joined #openstack-containers09:55
openstackgerritLingxian Kong proposed openstack/magnum master: Improve floating_ip_enabled  https://review.openstack.org/64154709:58
*** ykarel|lunch is now known as ykarel10:07
*** udesale has quit IRC10:34
strigaziflwang1: hello10:49
strigaziflwang1: are you here?10:49
flwang1yes10:49
strigazitell me10:49
flwang1i think lingxian has discussed with you about the floating ip design, are you happy with that?10:50
strigazilook ok10:50
flwang1nice10:51
flwang1strigazi: and here is the NPD patch https://review.openstack.org/64190210:51
strigazii have resize and NPD for today10:51
flwang1nice10:52
strigaziFYI I'll push a path to disable the CI10:52
strigaziwe should not burn resources10:52
flwang1oh, sure, we have to disable it for now10:52
flwang1we just need to be more careful10:52
openstackgerritFeilong Wang proposed openstack/magnum master: (WIP) Add cluster upgrade to the API  https://review.openstack.org/51495910:53
strigaziflwang1: https://ci.centos.org/artifacts/fedora-coreos/prod/builds/latest/10:54
strigazihttps://ci.centos.org/artifacts/fedora-coreos/prod/builds/latest/fedora-coreos-29.753-openstack.qcow2.gz10:54
flwang1wow10:54
flwang1do you have plan to move?10:55
flwang1https://review.openstack.org/#/c/514959/10/magnum/drivers/k8s_fedora_atomic_v1/driver.py@58 here is my design about the upgrade api10:55
strigaziflwang1: do you mind if we set a time for tmr morning you, (evening today for me) to discuss?10:55
flwang1strigazi: sure10:55
flwang1strigazi: it would be nice if you can take a look then we can discuss10:56
flwang19 utc?10:56
strigaziI will10:56
flwang121 i mean10:56
strigazithis https://www.timeanddate.com/worldclock/fixedtime.html?msg=chat&iso=20190311T2100 ?10:57
flwang1yep, is that ok for you?10:57
strigaziyeap10:57
flwang1cool10:57
strigazisee you then10:57
flwang1cool10:57
strigaziI have to go, I'll take a look in the patches10:58
flwang1thanks10:58
*** ricolin has quit IRC10:59
*** dave-mccowan has joined #openstack-containers11:10
*** ttsiouts has quit IRC11:12
*** dave-mccowan has quit IRC11:14
*** dave-mccowan has joined #openstack-containers11:20
*** ykarel_ has joined #openstack-containers11:22
*** ykarel has quit IRC11:24
*** ivve has quit IRC11:31
*** dave-mccowan has quit IRC11:42
*** ykarel_ is now known as ykarel11:49
*** dave-mccowan has joined #openstack-containers11:56
*** jchhatbar has quit IRC11:58
*** ttsiouts has joined #openstack-containers12:02
*** ivve has joined #openstack-containers12:14
*** janki has joined #openstack-containers12:35
*** ianychoi has quit IRC13:28
*** ianychoi has joined #openstack-containers13:29
*** sapd1 has joined #openstack-containers13:31
*** dave-mccowan has quit IRC13:33
*** henriqueof has quit IRC13:41
*** lxkong has quit IRC13:42
*** henriqueof has joined #openstack-containers14:05
*** dave-mccowan has joined #openstack-containers14:20
*** alisanhaji has quit IRC14:30
*** mkuf has quit IRC14:31
*** FlorianFa has joined #openstack-containers14:50
*** FlorianFa has quit IRC14:52
*** FlorianFa has joined #openstack-containers14:52
*** udesale has joined #openstack-containers14:56
*** sapd1 has quit IRC15:05
*** alisanhaji has joined #openstack-containers15:22
*** ykarel is now known as ykarel|afk15:28
*** janki has quit IRC15:33
*** flwang1 has quit IRC15:42
*** ramishra has quit IRC16:02
*** gsimondo1 has quit IRC16:06
*** ykarel_ has joined #openstack-containers16:07
*** ykarel|afk has quit IRC16:09
*** pcaruana has quit IRC16:23
*** pcaruana has joined #openstack-containers16:23
*** ykarel_ is now known as ykarel16:23
*** ykarel is now known as ykarel|away16:29
*** ivve has quit IRC16:32
*** udesale has quit IRC16:52
*** ttsiouts has quit IRC17:24
*** ykarel|away has quit IRC17:29
*** ivve has joined #openstack-containers17:35
*** hongbin has joined #openstack-containers17:40
*** flwang has joined #openstack-containers19:44
openstackgerritLingxian Kong proposed openstack/magnum master: Improve floating_ip_enabled  https://review.openstack.org/64154719:47
flwangNobodyCam: i don't think you can access the atomic container, why do you want to do that?19:54
*** lxkong has joined #openstack-containers20:00
*** itlinux has joined #openstack-containers20:30
*** itlinux has quit IRC20:34
*** itlinux has joined #openstack-containers20:56
strigaziNobodyCam runc exec -t kube-apiserver bash21:02
strigaziflwang: hey, I'm testing the patches21:02
strigaziflwang: Will get back to you in a bit21:02
flwangstrigazi: sure21:04
flwangi'm around, just let me know when you're ready to catch up21:04
*** henriqueof has quit IRC21:08
NobodyCamstrigazi: Awesome thank you :)21:13
NobodyCamcan some help me set a trusted root ca for Kube-controler-manager?21:14
NobodyCamI'm getting : `controllermanager.go:169] error building controller context: cloud provider could not be initialized: could not init cloud provider "openstack": Post https://10.21.100.198:5000/v3/auth/tokens: x509: certificate signed by unknown authority`21:14
NobodyCamHow ever I not sure what Need to be set to trust my cert21:15
strigazihttps://docs.openstack.org/magnum/latest/configuration/sample-config.html21:16
strigazi# Path to the OpenStack CA-bundle file to pass and install in all cluster21:16
strigazi# nodes. (string value)21:16
strigazi#openstack_ca_file =21:16
eanderssonstrigazi, flwang lets get this merged https://review.openstack.org/#/c/619148/21:19
eanderssonclean up, but a good one imo21:19
strigazi+221:21
strigaziflwang: we missed that https://github.com/helm/charts/tree/master/stable/node-problem-detector21:21
flwangstrigazi: you want to install with helm?21:22
NobodyCamstrigazi: Again Thank you :) testing that now21:22
flwangNobodyCam: better setup a devstack env and then you can compare the config21:23
strigazinow it is done, we could have less code, mainly helm helps on that21:24
*** alisanhaji has quit IRC21:24
strigaziNPD patch looks ok, as you want21:25
flwangstrigazi: we should have a policy to encourage using helm if we think that's the direction21:27
flwangstrigazi: i mean some documents for developer about this addons implementation21:28
strigaziWhat do you think?21:30
strigaziDocs are helpful, and we should tell developers how to add addons.21:32
flwangstrigazi: you mean do the NPD with helm now?21:32
eanderssonflwang, strigazi lets get this one in as well https://review.openstack.org/#/c/637266/21:34
eanderssonpy3.7 test job21:34
eanderssonhttps://review.openstack.org/#/c/637267/21:35
*** dave-mccowan has quit IRC21:36
flwangeandersson: thank you for all the good work21:36
eanderssonAnytime flwang21:36
eanderssonIt's good to keep the review backlog clean so we can focus on the important stuff.21:36
flwangeandersson: true21:37
strigaziI would prefer it this way. If you are up for it, you can change it. I don't want to tell you to do it again :)21:38
flwangstrigazi: thanks, let's get it in this time. and we will pay more attention next time about how to choose the right way for new addons, agree?21:41
openstackgerritSpyros Trigazis proposed openstack/magnum master: Support <ClusterID>/actions/resize API  https://review.openstack.org/63857221:43
strigaziI rebased to test resize we vm uuid21:44
strigaziok21:44
*** pcaruana has quit IRC21:45
*** ivve has quit IRC21:54
*** itlinux has quit IRC21:55
flwangstrigazi: did you get a chance to review the upgrade api patch?21:56
flwangstrigazi: still around?22:01
strigaziflwang: I'm testing resize now. I see you remove the helper script for testing upgrade. You don't test?22:17
openstackgerritMerged openstack/magnum master: Add missing ws separator between words  https://review.openstack.org/61914822:17
strigaziflwang: oh, you put it in an rst22:18
strigaziflwang: resize gives me 40522:18
strigaziflwang: 202 finally, but I passed as nodegroup: "error" and it worked22:21
flwangstrigazi: since i didn't use the nodegroup parameter now since we don't support it yet22:22
strigaziflwang: resize for scale up, is it working?22:23
strigaziflwang: works, but I need to pass empty list22:24
flwangstrigazi: yep, scale up works as well. we could set a default value for nodes_to_remove22:25
strigaziscale up while UPDATE_IN_PROGRESS returnes 202 but has no effect22:26
flwangso you think 204 could be better?22:27
strigaziif it has no effect, shouldn't return 4xx ?22:27
strigazior we can make to have an effect22:27
flwangwhat do you mean 'no effect' then?22:27
strigaziresize from 2 to 3 -> 202 and heat receives the request22:28
flwangsince the stack/cluster is "UPDATE_IN_PROGRESS", so i think it's effecting, isn't it?22:28
strigaziright after resize from 3 to 4 -> 202 but heat doesn't  receive the request22:28
flwangah, i think it's because we can't update a cluster which is in 'update_in_progress'22:29
flwangyou should get a warning https://review.openstack.org/#/c/638572/16/magnum/conductor/handlers/cluster_conductor.py@19622:30
flwangstrigazi:  i will dig, this could be a bug22:30
flwangnormally, if the cluster is being updated, then user can't issue another update22:30
strigaziwe could make it work, heat accepts it22:31
flwangok22:31
flwangbut i don't think that's good user case anway22:31
strigaziit is what the autoscaler wants though22:32
flwanggood point, i can see the idea now22:33
flwangyep, the workload could increase very fast, and CA may send request very often22:34
flwangi will do more test about that case22:35
flwangand you're welcome to submit patch set if you can sort it faster22:35
strigazihave you seen the nodegroup patches?22:36
flwangyep, i saw that22:36
flwangit's on my list, i will start to test it this week22:37
strigaziresize is close, needs a bit more testing22:38
flwangis the NPD patch OK for you?22:38
strigaziyes, one nit, I left a comment22:39
openstackgerritFeilong Wang proposed openstack/magnum master: [fedora-atomic-k8s] Adding Node Problem Detector  https://review.openstack.org/64190222:40
flwangcool, i just fixed it22:40
flwangstrigazi: let's discuss the rolling upgrade?22:40
flwangone problem when i testing the upgrade api, i found i can't get those default versions/tags if user didn't specified as labels22:42
flwangthat's OK, the only cons is we can't check the downgrade to raise 40022:42
strigazishall we make all the container tags another field?22:44
strigazior leave as a 2nd step22:45
strigaziresize seems to work22:45
strigazilet's check resize when already in progress and we are good22:45
flwangyou mean take the npd_tag off from current patch?22:46
flwangI will test the resize api again for the update-in-progress case22:47
flwangstrigazi: i can't see much benefit to put all tags into the 2nd patch22:49
flwanggiven we will merge them all finally in this release22:50
*** threestrands has joined #openstack-containers22:50
strigaziquestion about fips22:51
strigaziyou want to remove fips for all workers?22:51
strigaziI didn't realize that22:51
strigaziin our cloud, it doesn't matter22:51
strigazibut for other sites it seem extremly radical22:51
flwangstrigazi: we'd like to encourage customer to expose services via lb22:52
flwangand user can bind FIP for worker manually22:52
strigaziwasn't it an option to have fips master_only, not fips, and all?22:52
flwangstrigazi: that's the thing i discussed with lingxian, and we22:52
flwanglike to get comments from the community22:52
strigaziit is best practice22:53
lxkonghi strigazi22:53
strigazibut when people upgrade it is very unexpected22:53
flwangstrigazi: yep, i know it's a bit aggressive22:53
strigaziagain, for our cloud we don't care22:53
lxkongyeah, the patch removed the fip for workers. But users can still allocate fip for workers manually22:53
strigazibut for other might be really bad22:54
flwangeandersson: jakeyip: how do you guys think?22:54
lxkongstrigazi: that's why i sent an email to mailing list to gather feedback22:54
lxkongbut for now, no one replied22:54
eanderssonWe don't use this feature at the moment.22:54
lxkongi see Jim added himself to the reviewers, let's see how he will say22:54
strigazilxkong: flwang you really think it is a good move?22:55
eanderssonI added Jim22:55
lxkongstrigazi: i think so. We don't want to expose the workers to the public internet by default22:55
strigazithis is for your cloud ^^22:55
lxkongyeah22:55
lxkongso it's important to know if it affects otheres22:56
strigaziand an email for a couple of days will solve this?22:56
lxkongstrigazi: your suggestion?22:56
strigaziwhy not make the three options as we discussed in the past?22:56
eanderssonMy only concern is that while no one might use the "current' option today22:56
eanderssonbut what if someone installs Queens?22:56
eanderssonCould we back-port this behavior if we made the change?22:57
lxkongeandersson: in Queens, workers still have fip if floating_ip_enabled22:57
eanderssonSure - but what about when they upgrade.22:57
flwangstrigazi: what's the problem when you deal with this issue? i mean any blocker for your old patch to change the fip enabled from bool to enum?22:57
lxkongeandersson: what's your concern for upgrade?22:58
strigazithere is not blocker, I didn't have time and it was low priority22:58
flwangstrigazi: do you think a new label 'master_only' or something could be a balanced option?22:59
strigazicould be23:00
strigazimore friendly23:00
strigazifor catalyst, isn't even more economical not fips at all and only in the lbaas for the api?23:01
strigazifor catalyst, isn't even more economical no fips at all and only in the lbaas for the api?23:01
strigazilike this, it one fip per cluster23:02
flwangstrigazi: that's related to security23:02
strigazihow?23:02
lxkongstrigazi: the attach surface23:02
flwangreduced more exposure23:02
lxkongattack23:02
strigaziI don't think you understoop what I said. I was comparing23:03
strigazihaving fips in the master(s) nodes to having only a fip for the API LB23:03
strigaziin a cluster with three masters, you would have only the LB exposed, how is that less secure?23:04
strigaziand the most secure is zero fips23:04
lxkongthe lb only allows traffic to 6443 port23:04
strigaziwhich is bad?23:05
flwangstrigazi: user can also have 0 FIP23:05
lxkongstrigazi: but we need to allow the user to access the cluster from internet23:05
lxkongthe fip needs to be included in the api cerficiate23:05
strigaziit is23:05
lxkongif floating_ip_enabled is false, then the cluster is only for internal usage23:06
lxkongusers can never access the cluster even allocate fip to the vip later23:06
strigazihmm, the FIP of the LB is not23:06
strigaziso, if we add the LB FIP to the api cert we are good, no?23:07
lxkongwe are good23:07
strigaziisn't that even easier?23:07
lxkongany api could do this? or manually?23:07
strigaziI mean , when users asks for cluster with LB, the LB will have a fip23:08
lxkongno matter if 'floating_ip_enabled' is set or not?23:08
flwangi think that's not a bad idea23:09
strigazigood point, maybe add a label for that? so floating_ip_enabled controls only the fips on the vms23:09
lxkongmy proposal is, if floating_ip_enabled is set, only allocate fip to the lb vip, otherwise, no fip at all23:10
strigaziand for LB fip have a label?23:10
flwangstrigazi: i kind of like the ideea23:10
lxkongstrigazi: no a bad idea23:10
flwangstrigazi: lxkong: are we all good for that proposal?23:11
lxkongstrigazi: but if we should allocate fip for the master in the cluster only has one master?23:11
strigazithe proposal is good in theory, but from experience all openstack clouds are weird :)23:11
flwangstrigazi: hah, i won't argue that23:11
lxkonglb_fip label is true and floating_ip_enabled is false, how we deal with the cluster with only one master?23:12
strigaziyou can have single master cluster with LB? not much value on that but it works like this atm23:12
flwanglxkong: depends on if master lb is enabled23:12
lxkongah, yes23:12
flwanglxkong: if you have only one master, but you do want to enable lb, your call23:12
flwangeverything works as the same23:12
lxkongone master cluster can not have lb, right?23:13
strigaziat the moment it can23:13
flwanglxkong: it can if you enable the master_lb_enabled23:13
strigazi(it was like this because we were planning to scale up the masters)23:13
lxkongok, then all good23:13
lxkongwe add a label to control the lb vip23:14
lxkongleave floating_ip_enabled to control the vms23:14
strigazi+123:14
lxkongflwang:  agree?23:14
flwang+123:14
lxkongcool23:15
strigaziIMO, we should tell user to not expose the k8s api to the internet23:15
flwangstrigazi: still have time to discuss rolling upgrade? or you have to goto bed23:15
strigazithat is just suggestion, I agree with what we just said23:16
flwanglxkong: can you add a small document about that ^ ?23:16
lxkongi will23:16
flwanglxkong: thank you23:16
flwangstrigazi: i'm ok to let you go now ;) but you have to promise you will leave comments on https://review.openstack.org/#/c/514959/23:17
strigaziflwang: for upgrade, I think it is good to upgrade the addons/k8s-resources separately. We can achieve this with CTs that have only changes for tags23:18
strigaziI'll leave comments23:19
strigaziBefore going to bed, just smth to think about23:19
flwangstrigazi: cool. thank you very much. and I really appreciate for your time to stay late with us23:20
flwanghave a good night, my friend23:20
strigaziAt CERN, we have some custom plugings (four for storage and two for special site certs and kerberos keytabs)23:20
flwangso i think you are setting the tags explicitly?23:21
strigaziwe plan to write a helm chart for each one of those and then a meta-chart that will control all of them23:21
flwangthat would be cool23:21
strigazithis way we only release one chart that rules them all23:22
flwangwe could do that for magnum as well23:23
flwangin the future23:23
strigaziwe can do this upstream too, control all plugins with many charts and one meta chart which has only dependencies23:23
strigaziexactly23:23
flwangi'm happy to review it23:23
strigazicool23:23
flwangcool, i think we're all good for today?23:24
strigazisee you in the meeting tmr :) have a nice day lxkong flwang and evening eandersson23:24
strigaziflwang: yeap23:24
lxkonghave a good night23:25
strigazicheers23:25
eanderssonSorry had to step away for a bit23:31
eanderssonstrigazi, have a good one!23:31
*** threestrands_ has joined #openstack-containers23:42
*** threestrands has quit IRC23:45
*** hongbin has quit IRC23:57

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!