Tuesday, 2018-12-11

*** PagliaccisCloud has joined #openstack-containers00:52
*** hongbin has quit IRC01:00
*** ianychoi has quit IRC01:20
*** ricolin has joined #openstack-containers03:12
*** hongbin has joined #openstack-containers03:15
*** ykarel|away has joined #openstack-containers03:42
*** udesale has joined #openstack-containers03:48
*** ramishra has quit IRC03:59
*** ricolin has quit IRC04:03
*** udesale has quit IRC04:08
*** udesale has joined #openstack-containers04:43
*** lpetrut has joined #openstack-containers04:55
*** hongbin has quit IRC05:07
*** ykarel|away has quit IRC05:19
*** ramishra has joined #openstack-containers05:26
*** ykarel|away has joined #openstack-containers05:35
*** lpetrut has quit IRC05:35
*** ykarel|away is now known as ykarel05:42
*** sdake has quit IRC06:35
*** sdake has joined #openstack-containers06:40
*** rcernin has quit IRC06:43
*** mkuf_ has joined #openstack-containers06:58
*** mkuf has quit IRC07:03
*** ramishra has quit IRC07:15
*** ykarel is now known as ykarel|lunch07:31
*** mkuf_ has quit IRC07:52
*** mkuf_ has joined #openstack-containers07:54
*** ramishra has joined #openstack-containers08:01
*** mgoddard has quit IRC08:10
*** ykarel|lunch is now known as ykarel08:30
*** ramishra has quit IRC08:44
*** ramishra has joined #openstack-containers08:51
*** lpetrut has joined #openstack-containers09:21
openstackgerritBharat Kunwar proposed openstack/magnum stable/queens: k8s_fedora: Add cloud_provider_enabled label  https://review.openstack.org/62413209:29
*** shrasool has joined #openstack-containers09:56
*** salmankhan has joined #openstack-containers10:08
*** salmankhan has quit IRC10:21
*** salmankhan has joined #openstack-containers10:21
*** shrasool has quit IRC10:26
*** shrasool has joined #openstack-containers10:37
*** udesale has quit IRC10:56
*** tobias-urdin is now known as tobias-urdin|lun11:00
*** tobias-urdin|lun is now known as tobias-urdin_afk11:01
*** shrasool has quit IRC11:21
*** shrasool has joined #openstack-containers11:22
*** shrasool has quit IRC11:26
*** tobias-urdin_afk is now known as tobias-urdin11:27
*** salmankhan has quit IRC11:46
*** salmankhan has joined #openstack-containers11:50
*** mkuf has joined #openstack-containers12:26
*** mkuf_ has quit IRC12:30
*** dave-mccowan has joined #openstack-containers12:53
*** dave-mccowan has quit IRC13:01
*** ykarel is now known as ykarel|afk13:12
*** zul has quit IRC13:26
*** zul has joined #openstack-containers13:26
*** ykarel|afk has quit IRC13:57
*** salmankhan has quit IRC14:01
*** salmankhan has joined #openstack-containers14:01
*** mkuf_ has joined #openstack-containers14:06
*** dave-mccowan has joined #openstack-containers14:07
*** mkuf has quit IRC14:09
*** mkuf has joined #openstack-containers14:14
*** dave-mccowan has quit IRC14:14
brtknrmnaser: vexxhost is even involved in https://github.com/gophercloud/gophercloud14:14
brtknrawesome!14:14
*** mkuf_ has quit IRC14:17
*** shrasool has joined #openstack-containers14:18
*** ykarel|afk has joined #openstack-containers14:18
*** udesale has joined #openstack-containers14:33
*** ykarel|afk is now known as ykarel14:37
openstackgerritMerged openstack/magnum master: Add iptables -P FORWARD ACCEPT unit  https://review.openstack.org/61964314:42
*** salmankhan has quit IRC14:46
mnaserbrtknr: yep :)14:58
*** salmankhan has joined #openstack-containers14:59
brtknrlooks like cloud-controller-manager using gophercloud to interact with openstack API15:02
*** itlinux has quit IRC15:06
*** salmankhan1 has joined #openstack-containers15:20
*** salmankhan has quit IRC15:20
*** salmankhan1 is now known as salmankhan15:20
*** mkuf has quit IRC15:22
*** mkuf has joined #openstack-containers15:28
*** munimeha1 has joined #openstack-containers15:32
*** ivve has joined #openstack-containers15:37
*** ykarel is now known as ykarel|away15:38
*** munimeha1 has quit IRC16:11
*** udesale has quit IRC16:14
openstackgerritweizj proposed openstack/magnum master: Remove the space to keep docs consistence with others  https://review.openstack.org/59673416:19
*** hongbin has joined #openstack-containers16:19
*** itlinux has joined #openstack-containers16:22
*** mriedem has joined #openstack-containers16:52
mriedemlooking for some magnum cores to check out the upgrade-checkers community wide goal framework patch https://review.openstack.org/#/c/611505/16:52
mriedemhongbin: ^16:52
mriedemyou should be familiar b/c of the zun one16:52
hongbinmriedem: ack16:52
hongbinmriedem: lgtm16:54
mriedemthanks16:55
*** mriedem has left #openstack-containers16:57
*** itlinux_ has joined #openstack-containers16:59
*** itlinux has quit IRC17:03
*** zul has quit IRC17:20
*** ykarel|away has quit IRC17:35
*** salmankhan has quit IRC17:51
*** lpetrut has quit IRC17:56
*** shrasool_ has joined #openstack-containers18:43
*** shrasool has quit IRC18:45
*** shrasool_ is now known as shrasool18:45
*** lbragstad has quit IRC19:30
*** lbragstad has joined #openstack-containers19:31
*** munimeha1 has joined #openstack-containers19:39
*** zul has joined #openstack-containers20:05
*** shrasool has quit IRC20:34
*** shrasool has joined #openstack-containers20:35
openstackgerritSpyros Trigazis proposed openstack/magnum master: k8s_fedora: Use external kubernetes/cloud-provider-openstack  https://review.openstack.org/57747720:44
*** flwang has joined #openstack-containers20:48
flwangstrigazi: do we have meeting today?20:48
strigaziflwang: yes20:55
strigaziflwang: would this be useful? https://indico.cern.ch/category/10892/20:55
strigaziflwang: I can send automatic email too, indico is an open-source tool developed at CERN that we use to manage meeting, conferences, lectures etc20:56
flwangstrigazi: nice20:58
strigazi#startmeeting containers21:00
openstackMeeting started Tue Dec 11 21:00:29 2018 UTC and is due to finish in 60 minutes.  The chair is strigazi. Information about MeetBot at http://wiki.debian.org/MeetBot.21:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.21:00
*** openstack changes topic to " (Meeting topic: containers)"21:00
strigazi#topic Roll Call21:00
openstackThe meeting name has been set to 'containers'21:00
*** openstack changes topic to "Roll Call (Meeting topic: containers)"21:00
strigazio/21:00
cbrumm_o/21:00
flwango/21:01
strigazi#topic Announcements21:02
*** openstack changes topic to "Announcements (Meeting topic: containers)"21:02
strigaziNone21:03
strigazi#topic Stories/Tasks21:03
*** openstack changes topic to "Stories/Tasks (Meeting topic: containers)"21:03
strigaziflwang: mnaser pushed some patches to impove the CI21:03
strigazi#link https://review.openstack.org/#/q/topic:k8s-speed+(status:open+OR+status:merged)21:03
strigazithe last five is kind of refactor, the first ones are functional tests only.21:05
strigaziWith nested virt, the k8s test runs in ~40 min21:05
flwangvery cool21:05
flwangi'm keen to review them21:05
strigaziThey are small actually, thanks21:06
flwangstrigazi: 40 is ok for now, we can improve it later21:06
strigazi~30 takes only to deploy devstack21:06
flwangthat's quite reasonable then21:07
flwangi love then21:08
flwangin the future, we probably can support sonobuoy21:08
strigaziFrom my side I completed the patch to use the cloud-provider-openstack k8s_fedora: Use external kubernetes/cloud-provider-openstack  https://review.openstack.org/57747721:08
strigaziI did it because I was testing lxkong patch for the delete hooks.21:09
flwangstrigazi: yep, i saw that, i just completed the review21:10
strigazicbrumm_: you are using this provider https://hub.docker.com/r/k8scloudprovider/openstack-cloud-controller-manager/ or the upstream from kubernetes ?21:11
flwangcommented21:11
cbrumm_looks good, we might have some small additions to make to the cloud controller so it will be good to have those in21:11
cbrumm_we're using the k8s upstream21:11
strigazicbrumm_: with with k8s release?21:11
strigazicbrumm_: with which k8s release?21:11
cbrumm_yeah, matched to the same version of k8s, so 1.12 for right now21:12
strigaziI use the out-of-tree and out of k8s repo one. I think new changes go there, did I got this wrong?21:13
cbrumm_we use this one https://github.com/kubernetes/cloud-provider-openstack21:14
*** shrasool has quit IRC21:14
strigazioh, ok, my patch is using that one, cool21:14
cbrumm_good, that one is where all the work is going according to the slack channel21:15
strigazithat's it from me, flwang cbrumm_ anything you want to bring up?21:16
cbrumm_Nothing, I'm missing people at kubecon and paternity leave21:17
flwangstrigazi: the keystone auth patch is ready and in good shape in my standard21:17
flwangmay need small polish, need your comments21:17
strigaziThis kubecon in particural must be very cool21:17
strigaziflwang: I'm taking a look21:18
flwangstrigazi: and i'd like to discuss the rolling upgrade and auto healing if you have time21:18
strigaziflwang: the patch is missing only the tag (?). I just need to test it.21:20
flwangwhich patch?21:21
strigazithe keystone-auth one21:22
flwangyep, we probably need a label for its tag21:22
flwanghttps://hub.docker.com/r/k8scloudprovider/k8s-keystone-auth/tags/21:24
strigaziFor upgrades, I'm finishing what we discussed two weeks ago. I was a little busy with the CVE and some downstream ticketing. It's going well21:24
flwangcool21:24
flwangdoes the upgrade support master nodes?21:25
strigazibtw, I did this script to sync containers and create cluster fast http://paste.openstack.org/raw/737052/21:25
strigaziflwang: yes, with rebuild (if using a volume for etcd) and inplace21:26
flwangnice, thanks for sharing21:26
*** shrasool has joined #openstack-containers21:26
strigaziflwang: when moving to minor releases inplace should be ok.21:27
strigazi1.11.5 to 1.11.6 for example21:27
flwangi'm happy with current upgrade desgin21:27
flwangdo you want to discuss auto healing in meeting or offline?21:27
strigazilet's do it now, he have an 1 hour meeting anyway21:28
strigazibased on our experience with the k8s CVE21:28
strigazithe healing and monitoting of the heatlth can be two separate things that are both required.21:29
strigaziTo clarify:21:29
flwangi'm listening21:29
strigazimonitoting the health status or even version? os useful for an operator21:29
strigaziFor the CVE I had to cook a script that checks all apis if they allow anonymous-auth21:30
strigazi*all cluster APIs in our cloud.21:30
flwangos useful for an operator  ??  os=?21:30
strigazis/os/it is/21:31
*** shrasool has quit IRC21:32
strigaziFor autohealing, we could use the cluster autoscaler with node-detector and draino instead of writing a magnum specific authealing mechanism21:33
flwanganybody working on autohealing now? i'm testing node problem detector and draino now21:35
strigazidoes this make sense?21:35
cbrumm_node problem detector is something we'll be taking a deeper look at in Jan21:36
flwangfor health status monitoring, do you mean we still prefer to keep the periodic job to check the health status in Magnum?21:36
strigaziyes21:36
flwangcbrumm_: cool, please join us to avoid duplicating effort21:36
flwangstrigazi: ok, good21:37
lxkongstrigazi: there are 2 levels for auto-healing, the openstack infra and the k8s, and also we need to take care of both masters and workers21:37
cbrumm_sure, first we'll just be looking at the raw open source product, we aren't far at all21:37
lxkongi discussed auto-healing with flwang  yesterday, we have a plan21:38
strigaziwhat is the plan?21:39
strigaziwhat is openstack infra a different level?21:40
lxkonguse aodh and heat to guarantee the desired node count, NPD/drainer/autoscaler for the node problems of workers21:42
strigazifyi, flwang lxkong if we make aodh a dependency, at least at CERN we will diverge, we just decommisioned ceilometer21:43
lxkongbut we need some component to revieve notification and trigger alarm action21:44
lxkongeither it's ceilometer or aodh or something else21:44
strigazitwo years ago we stopped collecting metrics with ceilometers.21:44
strigazitwo years ago we stopped collecting metrics with ceilometer.21:45
flwanglxkong: as we discussed21:45
strigazifor notifications we started using logstash21:45
strigaziI don't think aodh is a very appealing option for adoption21:45
flwangthe heat/aodh workflow is the infra/physical layer healing21:46
lxkongyep21:46
flwanglet's focus on the healing which can be detected by NPD21:46
flwangsince catalyst cloud doesn't have aodh right now21:46
strigaziis there smth that NPD can't detect?21:46
flwangstrigazi: if the worker node lost connection or down suddenly, i don't think NPD have chance to detect21:47
lxkongif the node on which NPD is running dies21:47
cbrumm_It can detect whatever you make a check for. It can use "nagios" style plugins21:47
flwangsince it's running as a pod on that kubelet, correct me if i'm wrong21:47
strigaziflwang:  if the node stopped reporting for X time it can be considered for removal21:48
flwangby who?21:48
strigaziautoscaler21:49
flwangok, i haven't add autoscaler into my testing21:51
flwangif autoscaler can detect such kind of issue, then i'm ok with that21:51
strigazii couldn't find the link for it.21:52
flwangthat's alright21:52
strigaziI think that we can separate the openstack specific checks from the k8s ones.21:52
flwangat least, we're on the same page now21:52
strigaziaodh or other solutions can take care of nodes without the cluster noticing21:52
flwangNPD+draino+auotscaler     and    keep the health monitoring in magnum in parallel21:52
strigaziyes21:53
flwanghow autoscaler deal with openstack?21:53
flwanghow autoscaler tell magnum i want to replace a node?21:53
*** schaney has joined #openstack-containers21:54
strigazithe current options is talking to heat directly, but we work on the nodegroups implementation and a method that the magnum api will expose a node removal api.21:54
flwangstrigazi: great, "the magnum api will expose a node removal api", we discussed this yesterday, we probabaly need an api like 'openstack coe cluster replace <cluster ID> --node-ip xxx  /  --node-name yyy21:56
*** itlinux_ has quit IRC21:57
flwangit would be nice if autoscaler can talk to magnum instead of heat because it would be easy for the auto scaling case21:57
flwangfor auto scaling scenario, magnum needs to know the number of master/worker nodes21:57
flwangif autoscaler talks to heat dierectly, magnum can't know that info21:58
strigaziin the autoscaler implementation we test here: https://github.com/cernops/autoscaler/pull/3 the autoscaler talks to heat and then to magnum to update the node count.21:58
*** rcernin has joined #openstack-containers21:59
flwangis it testable now?21:59
strigaziyes, it is still WIP but it works22:00
flwangfantastic22:02
flwangwho is the right person i should talk to ? are you working on that?22:02
strigaziflwang: you can create an issue on github22:02
flwanggood idea22:03
strigaziit is me tghartland and ricardo22:03
schaneyo/ sorry I missed the meeting, cbrumm sent me some of the conversation though.  Was there any timeline on the nodegroup implementation in Magnum?22:03
strigaziyou can create one under cernops for now and then we can move things to k/autosclaer22:03
flwangvery cool, does that need any change in gophercloud?22:03
strigazischaney: stein and the sooner the better22:04
schaneygohpercloud will need the Magnum API updates integrated once that's complete22:04
strigaziflwang: Thomas (tghartland) updated the deps for gophercloud, when we have new magnum APIs we will have to push to gophercloud changes too22:05
schaneyawesome22:06
strigazischaney: you work on gophercloud? autosclaler? k8s? openstack? all the above?:)22:06
schaney=) yep! hoping to be able to help polish the autoscaler once everything is released, we sort of went a different direction in the mean time (heat only)22:08
strigazicool, of you have input, just shoot in github, we work there to be easy for people to give input22:09
strigazicool, if you have input, just shoot in github, we work there to be easy for people to give input22:09
flwangschaney: are you the young man we meet at Berlin?22:09
schaneywill do!22:09
flwangstrigazi: who is working on the magnum api change?22:09
schaney@flwang I wasn't in Berlin. no, maybe Duc was there?22:10
flwangis there a story to track that?22:10
flwangschaney: no worries22:10
flwangappreciate for your contribution22:10
strigazifor nodegroups it is Theodoros https://review.openstack.org/#/q/owner:theodoros.tsioutsias%2540cern.ch+status:open+project:openstack/magnum22:11
flwangfor the node group  feature,  how the autoscaler call it to replace a single node?22:13
flwangi think we need a place to discuss the overall design22:13
strigaziFor node-removal we need to document it explicitly. We don't have something.22:13
strigaziI'll create a new story, I'll push a spec too.22:14
strigazibased on discussions that we had in my team, the options are two:22:14
strigazione is to have an endpoint to delete a node from the cluster without passing the NG. The other is to remove a node from a NG explicitly.22:15
strigaziWe might need both actually, I'll write the spec, unless someone is really keen on writing it. :)22:16
strigazi@all Shall we end the meeting?22:17
flwangstrigazi: let's end it22:17
schaneysure yeah, thanks!22:17
strigaziThanks eveyone22:17
strigazi#endmeeting22:17
*** openstack changes topic to "OpenStack Containers Team"22:17
openstackMeeting ended Tue Dec 11 22:17:44 2018 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)22:17
openstackMinutes:        http://eavesdrop.openstack.org/meetings/containers/2018/containers.2018-12-11-21.00.html22:17
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/containers/2018/containers.2018-12-11-21.00.txt22:17
openstackLog:            http://eavesdrop.openstack.org/meetings/containers/2018/containers.2018-12-11-21.00.log.html22:17
lxkongstrigazi: is now a good time for you to chat about the cluster pre-delete patch? It's per coe because the resource deletion is related to coe instead of drivers, e.g. there is no difference for k8s resource cleanup on fedora and coreos.22:18
lxkongi didn't fully understand your concern22:18
flwangstrigazi: Catalyst Cloud is keen to support auto healing by Feb 2019, so i'm really interested in a solution which not hardly depending with node group22:18
strigaziwe don't have to have nodegroups, the autoscaler has the notion of NGs but for magnum we assume a single NG22:19
strigazilxkong: fedora and coreos will become one and in practice the coreos driver is not used unless someone had forked it.22:20
strigazilxkong: I would prefer to not have one more plugin mechanism and make the hook just a module in drivers.22:21
lxkongstrigazi: but different deployer may have different clean up requirment, if 'a module in drivers' can do that?22:22
strigazilxkong: if downstream someone wants a different implementation they can use the drivers plugin architecture to have specific changes22:22
strigazilxkong: different deployments can have different drivers, that is the point of having drivers22:23
lxkongstrigazi: do you mean create a folder under drivers in the repo?22:23
strigazithe drivers are not that big, it is easier to fork a driver and these specific pre delete hooks22:24
strigazilxkong: yes22:24
strigazilxkong: this folder will contain the pre-delete functionality.22:24
lxkongstrigazi: it's very hard to have a reference implementation for the pre-delete, so you mean, if we have the requirment, we just create a folder by ourselves and don't need to have it upstream?22:25
*** itlinux has joined #openstack-containers22:26
strigazilet's narrow this down, does catalyst have a special dependency that can't go upstream?22:26
lxkonge.g. we use octavia and we've patched kubelet22:26
lxkongso that lb create can have a description including the cluster uuid22:26
lxkongbut for others, that may not true22:27
lxkongalso when deleting the cluster, we may not delete the cinder volumes for the pv, but maybe not the case for the others22:27
strigazithat's ok, when my patch is in for the ccm, everyone will have the cluster id in the description22:28
lxkongwhat if someone is using neutron-lbaas?22:28
lxkongalso what  if someone like us are not using ccm?22:28
lxkongi remember yesterday there's a guy said octavia is not a option for them :-(22:30
strigaziisn't neutron lbaas deprecated?22:31
lxkongyes, but i think many people are still using that22:31
lxkongi also have concern that maybe in future, there are more integration between openstack and ccm22:32
lxkonge..g barbican22:32
strigazilxkong: my point is that clouds that diverged from the upstream project can write their own hooks22:32
strigaziusing the driver plugin22:32
lxkongyeah, i understand your concern22:33
lxkongso do you think it's acceptable for my lb clean up implementation but move it to a package under derivers?22:35
flwangstrigazi: is your ccm patch backport-able? i mean mind me cherrypicking it after it merged?22:35
strigaziflwang: it is, sure22:35
lxkongs/derivers/drivers22:35
flwangstrigazi: cool22:35
strigazilxkong: I think we can cover, octavia as the first pre-deletion hook. As a next step, for storage we can have a configurable option to purge volumes.22:37
strigazilxkong: if people ask for more options we can implement those or ask them to contribute.22:38
strigazilxkong: the design of drivers was chosen, to version OS, COE and plugins as a unit22:39
lxkongstrigazi: ok, sounds good. I still not sure about where i should put the code, e.g. we care about k8s_fedora_atomic, so is it correct the code should live in magnum/drivers/k8s_fedora_atomic_v1/driver.py?22:41
lxkongor a new folder called something like 'k8s_fedora_atomic_resource_cleanup_v1/driver.py'22:42
lxkongthe name looks ugly22:42
flwanglxkong: strigazi: i think the general framework could be in the heat folder, but the octavia hook can be placed in to fedora_atomic folder, unless other user want to port it to other OS drivers22:44
*** itlinux has quit IRC22:45
lxkongflwang: that's also the way i prefer, but wanna double check with strigazi22:46
lxkongbecause that will affect all users who are using fedora_atomic22:46
strigaziwe can put it in the heat folder22:47
strigaziif there the cloud doesn't have octavia the hook will just pass right?22:47
strigaziif the cloud doesn't have octavia the hook will just pass right?22:47
lxkongstrigazi: i can add that check22:48
strigazilxkong: I think it is the same trick as you did when octavia is enabled or not.22:48
flwangstrigazi: it should be, as I talked with lxkong, the design of hook shouldn't impact any current logic22:48
lxkongstrigazi: yes22:49
lxkongstrigazi: cool, all clear now, thanks for your time22:49
lxkongi will udpate the patch accordingly22:49
strigazilxkong: cool22:49
strigaziflwang: let's take mnaser's changes, I'm thrilled to have the k8s CI working again.22:50
strigazihttps://review.openstack.org/#/q/status:open+project:openstack/magnum+branch:master+topic:k8s-speed22:50
flwangstrigazi: no problem22:50
lxkongstrigazi, flwang: i've tested this patch https://review.openstack.org/#/c/623724/, but the `OS::Heat::SoftwareDeploymentGroup` doens't work in my devstack22:51
lxkongthat resource hangs until timeout22:51
lxkongand heat-container-agent didn't get anything22:51
strigazilxkong: not this patch22:52
strigazilxkong: only the ones I +2, this patch needs work22:52
lxkongit's not the functional ones22:52
lxkonganyway, i will talk to mnaser, that's the one i'm interested we will see how much time it could save for clutser creation22:53
strigaziI'm going to sleep guys, thanks for all the work, it is great working together22:54
lxkongstrigazi: have a good night22:54
*** shrasool has joined #openstack-containers22:55
*** hongbin has quit IRC23:34
*** hongbin has joined #openstack-containers23:35
openstackgerritMerged openstack/magnum master: functional: retrieve cluster to get stack_id  https://review.openstack.org/62357523:51
*** hongbin has quit IRC23:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!