| *** livelace2 has quit IRC | 00:10 | |
| *** livelace2 has joined #openstack-containers | 00:10 | |
| *** shu-mutow has joined #openstack-containers | 00:12 | |
| *** itlinux has quit IRC | 00:13 | |
| *** threestrands has quit IRC | 00:38 | |
| *** hongbin has joined #openstack-containers | 00:39 | |
| openstackgerrit | Feilong Wang proposed openstack/magnum master: Add README.rst and Makefile for heat-container-agent https://review.openstack.org/585061 | 00:44 |
|---|---|---|
| *** rcernin_ has joined #openstack-containers | 00:56 | |
| *** rcernin has quit IRC | 00:58 | |
| *** yamamoto has joined #openstack-containers | 01:20 | |
| *** yamamoto has quit IRC | 01:24 | |
| *** Bhujay has joined #openstack-containers | 01:24 | |
| *** ricolin has joined #openstack-containers | 01:43 | |
| *** armaan has quit IRC | 02:19 | |
| *** armaan has joined #openstack-containers | 02:20 | |
| *** jmlowe has joined #openstack-containers | 02:23 | |
| *** dave-mccowan has quit IRC | 02:31 | |
| *** ramishra has joined #openstack-containers | 03:01 | |
| *** vijaykc4 has joined #openstack-containers | 03:04 | |
| *** Bhujay has quit IRC | 03:17 | |
| *** pengdake has joined #openstack-containers | 03:26 | |
| *** adrianreza has quit IRC | 03:34 | |
| *** vijaykc4 has quit IRC | 03:38 | |
| *** jmlowe has quit IRC | 03:43 | |
| *** lpetrut has joined #openstack-containers | 03:53 | |
| *** mdnadeem has joined #openstack-containers | 03:58 | |
| *** ykarel has joined #openstack-containers | 04:03 | |
| *** armaan has quit IRC | 04:15 | |
| *** hongbin has quit IRC | 04:16 | |
| *** armaan has joined #openstack-containers | 04:18 | |
| *** pengdake has quit IRC | 04:27 | |
| *** lpetrut has quit IRC | 04:42 | |
| *** vijaykc4 has joined #openstack-containers | 05:00 | |
| *** yasemin has quit IRC | 05:02 | |
| *** armaan has quit IRC | 05:05 | |
| *** armaan has joined #openstack-containers | 05:06 | |
| *** armaan has quit IRC | 05:09 | |
| *** pc_m has quit IRC | 05:09 | |
| *** armaan has joined #openstack-containers | 05:10 | |
| *** pc_m has joined #openstack-containers | 05:13 | |
| *** vijaykc4 has quit IRC | 05:39 | |
| *** armaan has quit IRC | 05:43 | |
| *** armaan has joined #openstack-containers | 05:44 | |
| *** mjura has joined #openstack-containers | 05:44 | |
| *** yamamoto has joined #openstack-containers | 05:46 | |
| *** yamamoto has quit IRC | 05:48 | |
| *** yamamoto has joined #openstack-containers | 05:56 | |
| *** yamamoto has quit IRC | 06:00 | |
| *** adrianc_ has joined #openstack-containers | 06:00 | |
| *** adrianc__ has joined #openstack-containers | 06:01 | |
| *** adrianc_ has quit IRC | 06:05 | |
| *** janki has joined #openstack-containers | 06:06 | |
| *** yasemin has joined #openstack-containers | 06:08 | |
| *** adrianc__ has quit IRC | 06:20 | |
| *** gsimondon has joined #openstack-containers | 06:24 | |
| *** armaan_ has joined #openstack-containers | 06:26 | |
| *** armaan has quit IRC | 06:29 | |
| *** lpetrut has joined #openstack-containers | 06:33 | |
| *** pcaruana has joined #openstack-containers | 06:34 | |
| *** ispp has joined #openstack-containers | 07:05 | |
| *** udesale has joined #openstack-containers | 07:06 | |
| *** ykarel is now known as ykarel|lunch | 07:33 | |
| *** lpetrut has quit IRC | 07:48 | |
| *** AlexeyAbashkin has joined #openstack-containers | 07:50 | |
| *** mjura has quit IRC | 07:53 | |
| *** lpetrut has joined #openstack-containers | 07:55 | |
| *** Bhujay has joined #openstack-containers | 07:55 | |
| *** rcernin_ has quit IRC | 07:56 | |
| *** janki has quit IRC | 07:58 | |
| *** janki has joined #openstack-containers | 07:58 | |
| *** Bhujay has quit IRC | 08:07 | |
| openstackgerrit | Merged openstack/magnum master: Switch to stestr https://review.openstack.org/582100 | 08:09 |
| *** yamamoto has joined #openstack-containers | 08:09 | |
| *** rcernin_ has joined #openstack-containers | 08:10 | |
| *** shu-mutow has quit IRC | 08:18 | |
| openstackgerrit | Vu Cong Tuan proposed openstack/python-magnumclient master: Switch to stestr https://review.openstack.org/585168 | 08:19 |
| *** ykarel|lunch is now known as ykarel | 08:21 | |
| openstackgerrit | Vu Cong Tuan proposed openstack/magnum-tempest-plugin master: Switch to stestr https://review.openstack.org/585172 | 08:24 |
| *** yamamoto has quit IRC | 08:26 | |
| *** ispp has quit IRC | 08:30 | |
| *** armaan_ has quit IRC | 08:34 | |
| *** parasitid has quit IRC | 08:34 | |
| *** armaan has joined #openstack-containers | 08:34 | |
| *** ispp has joined #openstack-containers | 08:38 | |
| flwang1 | strigazi: do we have meeting today? | 08:40 |
| strigazi | yes | 08:41 |
| flwang1 | strigazi: ok, cool | 08:41 |
| openstackgerrit | Vu Cong Tuan proposed openstack/python-magnumclient master: Switch to stestr https://review.openstack.org/585168 | 08:42 |
| *** ricolin has quit IRC | 08:42 | |
| strigazi | flwang1: I faced a new first in openstack | 08:49 |
| strigazi | flwang1: works in production, doesn't work in devstack! | 08:49 |
| flwang1 | strigazi: what's the problem? | 08:51 |
| *** parasitid has joined #openstack-containers | 08:52 | |
| strigazi | flwang1: kubernetes v1.11.1 works in production but it doesn't in devstack, I get E0724 08:52:02.604927 1 authentication.go:62] Unable to authenticate the request due to an error: x509: certificate specifies an incompatible key usage on the api. | 08:52 |
| *** mjura has joined #openstack-containers | 08:56 | |
| *** vijaykc4 has joined #openstack-containers | 09:01 | |
| *** janki has quit IRC | 09:01 | |
| *** vijaykc4 has quit IRC | 09:05 | |
| *** vijaykc4 has joined #openstack-containers | 09:05 | |
| *** udesale has quit IRC | 09:06 | |
| *** vijaykc4 has quit IRC | 09:06 | |
| *** armaan has quit IRC | 09:07 | |
| *** armaan has joined #openstack-containers | 09:08 | |
| *** skyscraper has quit IRC | 09:09 | |
| *** skyscraper has joined #openstack-containers | 09:13 | |
| flwang1 | strigazi: any difference between your prod and devstack? | 09:16 |
| strigazi | flwang1: DNS | 09:16 |
| strigazi | flwang1: in prod we have dns | 09:17 |
| flwang1 | strigazi: ok, i will try the 1.11.1 image | 09:17 |
| flwang1 | does it work for 1.11.0? | 09:18 |
| strigazi | flwang1: I will try | 09:20 |
| strigazi | flwang1: it is the same image the 1.11.1 in prod and devstack | 09:21 |
| *** udesale has joined #openstack-containers | 09:28 | |
| *** vijaykc4 has joined #openstack-containers | 09:37 | |
| *** gsimondo1 has joined #openstack-containers | 09:42 | |
| *** gsimondon has quit IRC | 09:45 | |
| *** vijaykc4 has quit IRC | 09:49 | |
| *** vijaykc4 has joined #openstack-containers | 09:50 | |
| flwang1 | strigazi: ok, btw, should we followup the functional testing on openlab? | 09:51 |
| flwang1 | currently, without functional testing making things tricky | 09:52 |
| strigazi | flwang1 openlab testing would be a good idea, but I don't know what they want from us or how to communicate things | 09:54 |
| strigazi | flwang1: in this case it wouldn't help whatsoever | 09:54 |
| flwang1 | strigazi: what i'm thinking is, because we can't do it with current openstack infra, so we HAVE TO look after any other way to do that | 09:55 |
| strigazi | flwang1 I know, I spend months on this and I gave up | 09:55 |
| flwang1 | otherwise, we will be fxxked in the future | 09:55 |
| flwang1 | strigazi: ok, i will try and see what can i do | 09:56 |
| strigazi | no we won't, it just more difficult to accept changes | 09:56 |
| flwang1 | strigazi: ok, for example, if we have a working function test, shit like this https://review.openstack.org/#/c/584215/ won't happen | 09:57 |
| flwang1 | ok, i know it's not a good sample | 09:58 |
| flwang1 | but you got my point, we have too many corners need to be covered | 09:58 |
| strigazi | I have pinged hogepodge dims and mnaser about openlab testing. My feeling is and I'm extremely confident about that there is not space for magnum in openlab testing. | 09:59 |
| strigazi | I'm mentioning this ^^ because it is not overlooked by me. | 10:01 |
| strigazi | flwang1: 1.11.0 works :( | 10:01 |
| flwang1 | strigazi: ok, got it | 10:01 |
| strigazi | #startmeeting containers | 10:02 |
| openstack | Meeting started Tue Jul 24 10:02:01 2018 UTC and is due to finish in 60 minutes. The chair is strigazi. Information about MeetBot at http://wiki.debian.org/MeetBot. | 10:02 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 10:02 |
| *** openstack changes topic to " (Meeting topic: containers)" | 10:02 | |
| strigazi | #topic Roll Call | 10:02 |
| openstack | The meeting name has been set to 'containers' | 10:02 |
| *** openstack changes topic to "Roll Call (Meeting topic: containers)" | 10:02 | |
| flwang1 | strigazi: if so, there are probably some regression issues | 10:02 |
| flwang1 | we may need review the changes between 1.11.1 and 1.11.0 | 10:02 |
| flwang1 | o/ | 10:02 |
| strigazi | o/ | 10:02 |
| strigazi | #topic Blueprints/Bugs/Ideas | 10:03 |
| *** openstack changes topic to "Blueprints/Bugs/Ideas (Meeting topic: containers)" | 10:03 | |
| flwang1 | strigazi: seems there are always only you and me in this meeting | 10:03 |
| strigazi | #topic meeting time | 10:04 |
| flwang1 | maybe we can merge the two meetings | 10:04 |
| *** openstack changes topic to "meeting time (Meeting topic: containers)" | 10:04 | |
| slunkad | hi | 10:04 |
| strigazi | I could late night for me, early for you and normal for blizzard? | 10:05 |
| strigazi | slunkad: hello | 10:05 |
| *** sfilatov has joined #openstack-containers | 10:05 | |
| strigazi | flwang1: 2100 UTC ? | 10:06 |
| flwang1 | strigazi: 2100UTC means 10AM, IIRC | 10:07 |
| flwang1 | for NZ time | 10:07 |
| strigazi | 1400 for west coast | 10:07 |
| strigazi | 0900 for NZ | 10:08 |
| strigazi | 2300 Europe | 10:08 |
| strigazi | flwang1: thoughts? | 10:09 |
| flwang1 | works for me | 10:10 |
| strigazi | for europe I'm present in working hours | 10:10 |
| strigazi | I can setup something like office hours | 10:10 |
| strigazi | Tuesday mornings | 10:10 |
| flwang1 | strigazi: that would be nice | 10:10 |
| strigazi | So if someone wants something can find me for sure. | 10:10 |
| strigazi | IMO, for our case we should push things in th ML | 10:11 |
| strigazi | slunkad: what do you think about office hours? | 10:11 |
| flwang1 | totally agree, we can put more our discussion in the ML | 10:11 |
| strigazi | it is like a meeting but without minutes. it is still logged. | 10:12 |
| slunkad | strigazi: what do you mean like setting it up as your status on irc? | 10:12 |
| strigazi | no, in the wiki page I'll post that this time and day someone will be on the channel | 10:13 |
| strigazi | well for europe me | 10:14 |
| slunkad | ok ya that sounds good | 10:14 |
| flwang1 | strigazi: i can cover the NZ/AP time if it's helpful | 10:15 |
| strigazi | ok then, Tuesdays at 1300 UTC europe maybe east coast for me | 10:15 |
| strigazi | flwang1: for you? | 10:16 |
| strigazi | you can pick a time that you are online and it is day :) | 10:16 |
| strigazi | you can pick a time that you are online and it is daytime :) | 10:16 |
| flwang1 | strigazi: yep | 10:16 |
| slunkad | does that mean the thursday meeting will not happen? | 10:16 |
| flwang1 | probably My Thursday morning | 10:16 |
| strigazi | we can move the meeting on Tuesdays 2100 or 2200 UTC | 10:17 |
| strigazi | 2100 UTC is a go? | 10:18 |
| strigazi | flwang1: ^^ | 10:19 |
| flwang1 | strigazi: works for me | 10:19 |
| strigazi | #agreed meeting moves to Tuesdays 2100 UTC | 10:20 |
| strigazi | We can do it today | 10:20 |
| strigazi | tmr for you flwang1 | 10:20 |
| flwang1 | strigazi: sure | 10:21 |
| strigazi | Next week I'll be on holidays. We can still have a meeting and flwang1 chairs it? | 10:22 |
| flwang1 | strigazi: no problem | 10:22 |
| flwang1 | i will call you if there is question i can't answer | 10:22 |
| strigazi | :) | 10:22 |
| strigazi | office hours Tuesdays at 1300 UTC for me | 10:23 |
| *** vijaykc4 has quit IRC | 10:24 | |
| strigazi | flwang1: do you want to set office hours? | 10:24 |
| *** vijaykc4 has joined #openstack-containers | 10:24 | |
| flwang1 | strigazi: let me check the UTC time of mine | 10:24 |
| *** vijaykc4 has quit IRC | 10:25 | |
| *** vijaykc4 has joined #openstack-containers | 10:26 | |
| flwang1 | Wed UTC 10:00PM - 11:00PM | 10:26 |
| strigazi | pm? | 10:26 |
| flwang1 | PM means my AM, and it may work for others, like afternoon | 10:27 |
| flwang1 | i don't know | 10:27 |
| flwang1 | or i can put mine later | 10:28 |
| strigazi | oh, so 2200 UTc ok | 10:28 |
| strigazi | sounds good | 10:28 |
| strigazi | #agreed office hours for strigazi Tuesdays at 1300 UTC and Wednesdays 2200 UTC for flwang1 | 10:29 |
| flwang1 | cool | 10:30 |
| strigazi | #topic Blueprints/Bugs/Ideas | 10:30 |
| *** openstack changes topic to "Blueprints/Bugs/Ideas (Meeting topic: containers)" | 10:30 | |
| strigazi | For me, I'll push to finish the upgrade API to have it in rocky, server and client by Friday. flwang1 I'll need you help for reviews. | 10:31 |
| flwang1 | strigazi: no problem, i'm keen to review it | 10:31 |
| strigazi | The implementation will do inplace upgrades I haven't managed to do the replace with draining. | 10:33 |
| strigazi | And secondly, I'll investigate the issue with kube v1.11.1 | 10:34 |
| strigazi | v1.11.0 and v1.11.1 work at the CERN cloud and they pass the conformance tests. | 10:34 |
| strigazi | v1.11.0 works on devstack, but v1.11.1 doesn't | 10:35 |
| strigazi | There is an issue with RBAC or certs | 10:35 |
| strigazi | I might try a devstack with designate enabled. | 10:35 |
| *** vijaykc4 has quit IRC | 10:36 | |
| strigazi | The only big difference is that in t production we have DNS and authentication is done with the node names. | 10:36 |
| strigazi | node name == hostname | 10:36 |
| strigazi | kube node name == hostname == nova vm name | 10:37 |
| flwang1 | that's possible | 10:37 |
| strigazi | what is possible? | 10:37 |
| flwang1 | i mean maybe related to DNS | 10:38 |
| strigazi | that it is for th millionth time DNS? :) | 10:38 |
| *** vijaykc4 has joined #openstack-containers | 10:38 | |
| strigazi | that it is for the millionth time DNS? :) | 10:38 |
| flwang1 | no, maybe related the hostname something | 10:38 |
| strigazi | http://i.imgur.com/eAwdKEC.png | 10:38 |
| strigazi | I couldn't resist | 10:39 |
| strigazi | I'm investigating, it is good dive in how auth works in k8s | 10:39 |
| flwang1 | i like the paint, typical chinese paint | 10:39 |
| flwang1 | strigazi: yep, as for auth, did you ever put some effort on the best practice of k8s security? | 10:40 |
| flwang1 | since I didn't see there is a sig-security in k8s community | 10:40 |
| flwang1 | so i'm wondering if there is team caring about it | 10:40 |
| strigazi | AFAIK we are doing the best possible. apart from selinux... | 10:41 |
| strigazi | after adding calico we covered the policy part too. | 10:41 |
| flwang1 | selinux is another topic, i don't think we still need to disable it, right？ | 10:41 |
| strigazi | we could have a label | 10:42 |
| strigazi | if selinux is on | 10:42 |
| strigazi | user will need to modify their temlates with the appropriate labeling | 10:42 |
| strigazi | the security-context in the pod spec | 10:43 |
| strigazi | https://kubernetes.io/docs/setup/independent/install-kubeadm/ | 10:44 |
| strigazi | Disabling SELinux by running setenforce 0 is required to allow containers to access the host filesystem, which is required by pod networks for example. You have to do this until SELinux support is improved in the kubelet. | 10:44 |
| strigazi | I have by passed all the issues with selinux on but I'm not very confident on having it on | 10:45 |
| flwang1 | strigazi: ok, is kubeadm doing the same? | 10:46 |
| strigazi | yeap | 10:46 |
| strigazi | but it works with selinux on | 10:46 |
| *** sfilatov_ has joined #openstack-containers | 10:47 | |
| strigazi | we can follow this offline with help from #fedora | 10:47 |
| flwang1 | strigazi: ok, cool | 10:48 |
| strigazi | To conclude, the cluster in queens and master are secure. After that, it is on the cluster admin to deploy apps securely. | 10:48 |
| strigazi | that is it from me. | 10:49 |
| flwang1 | cool | 10:49 |
| sfilatov_ | I have a question: how do we cope with upgrades in case of one of the k8s manifests needs to be changed? | 10:49 |
| *** sfilatov has quit IRC | 10:50 | |
| sfilatov_ | For example when I upgraded k8s from 1.9.8 to 1.10 I faced an issue with grafana manifest | 10:50 |
| sfilatov_ | image version for grafana needs to be bumped | 10:50 |
| sfilatov_ | Will the new upgrade logic allow us to upgrade this version? | 10:51 |
| strigazi | versions are easy to bump | 10:51 |
| strigazi | whereever we have a label with the tag | 10:51 |
| flwang1 | strigazi: as long as we pass it as a label i think | 10:51 |
| strigazi | the agent will run apply and bump it. | 10:52 |
| flwang1 | but for grafana, IIRC, we're hardcode the version | 10:52 |
| *** rcernin_ has quit IRC | 10:52 | |
| strigazi | for grafana yes | 10:52 |
| sfilatov_ | So upgrade should support versions? | 10:52 |
| sfilatov_ | i mean labels* | 10:52 |
| strigazi | yes | 10:52 |
| strigazi | since kube_tag is a label | 10:53 |
| strigazi | the same applies for all | 10:53 |
| sfilatov_ | So the idea is to pass all the needed labels | 10:54 |
| sfilatov_ | openstack coe cluster upgrade <cluster name or id> \ | 10:54 |
| sfilatov_ | --masters \ | 10:54 |
| sfilatov_ | --rollback \ | 10:54 |
| sfilatov_ | --batch-size <size> \ | 10:54 |
| sfilatov_ | --parameters key1=val1,...,keyN=valN | 10:54 |
| sfilatov_ | like in this spec? | 10:54 |
| strigazi | sfilatov_: can you push a patch for the missing ones? | 10:54 |
| sfilatov_ | yes, I can do that | 10:55 |
| strigazi | coredns and the monitoing ones | 10:55 |
| sfilatov_ | OK | 10:55 |
| sfilatov_ | Also could you tell the status for cluster_healing and nodepools? | 10:56 |
| sfilatov_ | Looks like they are on hold | 10:56 |
| sfilatov_ | Are there any plans on getting back to them? | 10:57 |
| strigazi | we can have the moniting in (which is required for healing) | 10:57 |
| strigazi | nodegroups won't make it in Rocky | 10:57 |
| strigazi | for autohealing we can have it | 10:58 |
| sfilatov_ | didn't quite get it | 10:58 |
| sfilatov_ | we are talking about heat autohealing, right? | 10:58 |
| strigazi | no | 10:58 |
| strigazi | magnum will monitor the cluster nodes | 10:59 |
| sfilatov_ | is there a patch for this? | 10:59 |
| strigazi | this the 1st part, it will basically ask the api if the nodes are ok. | 10:59 |
| sfilatov_ | I haven't seen it | 10:59 |
| strigazi | flwang1: has one | 10:59 |
| sfilatov_ | Kubernetes API? | 10:59 |
| strigazi | yes | 10:59 |
| flwang1 | sfilatov_: wait a sec | 11:00 |
| flwang1 | https://review.openstack.org/570818 | 11:00 |
| flwang1 | https://review.openstack.org/572897 | 11:00 |
| sfilatov_ | I guess it's okay for the first part, but generally would be better to get notifications from it | 11:00 |
| sfilatov_ | flwang1: thx! | 11:00 |
| sfilatov_ | strigazi: >nodegroups won't make it in Rocky | 11:00 |
| flwang1 | sfilatov_: no problem | 11:00 |
| strigazi | sfilatov_: openstack notifications? | 11:00 |
| strigazi | sfilatov_: user notifications? | 11:01 |
| strigazi | these are veeery different things. | 11:01 |
| flwang1 | sfilatov_: i think magnum can send openstack notifications out for the status change | 11:01 |
| strigazi | flwang1: ^^ this easy and very doable | 11:01 |
| flwang1 | strigazi: yep | 11:02 |
| sfilatov_ | we have a lot of issues | 11:02 |
| strigazi | sfilatov_: we have someone that just started working again in nodegroups but there is space for more people. In any case nodegroups will be in the next release (Rocky) | 11:02 |
| strigazi | sfilatov_: what issues? | 11:03 |
| sfilatov_ | when K8s api is not available | 11:03 |
| sfilatov_ | for some reason | 11:03 |
| sfilatov_ | that's why I like it better when nodes send notifications | 11:03 |
| sfilatov_ | but I guess we can talk about it in patch set | 11:03 |
| strigazi | we can ask kubelet\ | 11:04 |
| strigazi | curl https://$KUBELET_IP:10250/healthz == 'ok' | 11:04 |
| sfilatov_ | yeah | 11:04 |
| strigazi | polling | 11:04 |
| sfilatov_ | I'll check the patches then | 11:04 |
| strigazi | not pushing | 11:04 |
| sfilatov_ | and will comment it offline | 11:04 |
| strigazi | ok | 11:05 |
| sfilatov_ | strigazi: And if you plan on adding push-upgrades | 11:05 |
| sfilatov_ | how do you implement master/minions? | 11:05 |
| sfilatov_ | I mean I thought you need some kind of nodepools for that | 11:05 |
| strigazi | I'll ping you in the review. | 11:05 |
| sfilatov_ | thx | 11:05 |
| strigazi | we have two now, master and minion | 11:05 |
| strigazi | Let's wrap them. I'll send a summary in the ML and cc you | 11:06 |
| flwang1 | strigazi: thanks | 11:06 |
| strigazi | We need to use the ML | 11:06 |
| sfilatov_ | thx | 11:06 |
| strigazi | before closing | 11:06 |
| flwang1 | currently, i just finished the multi region issue and mainly focus on our magnum deployment | 11:06 |
| strigazi | sfilatov_: flwang1 slunkad if tou have time, test v1.11.0 seems to work well | 11:07 |
| *** sfilatov_ has quit IRC | 11:07 | |
| strigazi | I'll continue on v1.11.1 | 11:07 |
| flwang1 | strigazi: it's on my list now | 11:07 |
| strigazi | flwang1: great news \o/ | 11:07 |
| flwang1 | strigazi: yep, i'm so excited about that | 11:07 |
| *** sfilatov has joined #openstack-containers | 11:07 | |
| strigazi | we are going multiregion here too, we'll need to talk | 11:08 |
| strigazi | let's wrap, we are 9 mins late | 11:08 |
| strigazi | ok? | 11:08 |
| strigazi | said once | 11:09 |
| *** sfilatov has quit IRC | 11:09 | |
| *** sfilatov has joined #openstack-containers | 11:09 | |
| strigazi | said twice | 11:09 |
| strigazi | Thanks for joining folks! | 11:09 |
| strigazi | #endmeeting | 11:10 |
| *** openstack changes topic to "OpenStack Containers Team" | 11:10 | |
| openstack | Meeting ended Tue Jul 24 11:10:03 2018 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 11:10 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/containers/2018/containers.2018-07-24-10.02.html | 11:10 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/containers/2018/containers.2018-07-24-10.02.txt | 11:10 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/containers/2018/containers.2018-07-24-10.02.log.html | 11:10 |
| *** sfilatov has quit IRC | 11:10 | |
| *** sfilatov has joined #openstack-containers | 11:10 | |
| *** yamamoto_ has joined #openstack-containers | 11:14 | |
| *** sfilatov has quit IRC | 11:14 | |
| *** sfilatov has joined #openstack-containers | 11:14 | |
| *** sfilatov has quit IRC | 11:14 | |
| *** sfilatov has joined #openstack-containers | 11:16 | |
| *** sfilatov has quit IRC | 11:16 | |
| *** yamamoto_ has quit IRC | 11:27 | |
| *** udesale has quit IRC | 11:28 | |
| *** sfilatov has joined #openstack-containers | 11:33 | |
| *** yasufum has joined #openstack-containers | 11:41 | |
| *** yasufum_ has joined #openstack-containers | 11:43 | |
| *** yasufum has quit IRC | 11:45 | |
| *** yasufum has joined #openstack-containers | 11:46 | |
| *** vijaykc4 has quit IRC | 11:47 | |
| *** yasufum_ has quit IRC | 11:48 | |
| *** serlex has joined #openstack-containers | 11:49 | |
| *** linkmark has joined #openstack-containers | 11:54 | |
| *** yasufum_ has joined #openstack-containers | 11:54 | |
| *** yasufum has quit IRC | 11:56 | |
| *** yasufum_ is now known as yasufum | 11:56 | |
| *** sfilatov_ has joined #openstack-containers | 11:58 | |
| *** sfilatov has quit IRC | 11:58 | |
| *** vijaykc4 has joined #openstack-containers | 12:00 | |
| *** sfilatov has joined #openstack-containers | 12:00 | |
| *** sfilatov_ has quit IRC | 12:03 | |
| *** yasufum has quit IRC | 12:04 | |
| *** strigazi has quit IRC | 12:13 | |
| *** strigazi has joined #openstack-containers | 12:14 | |
| *** armaan has quit IRC | 12:26 | |
| *** armaan has joined #openstack-containers | 12:26 | |
| *** ispp has quit IRC | 12:39 | |
| *** ispp has joined #openstack-containers | 12:59 | |
| *** flwang1 has quit IRC | 13:05 | |
| *** flwang1 has joined #openstack-containers | 13:06 | |
| *** ianychoi has quit IRC | 13:08 | |
| *** ianychoi has joined #openstack-containers | 13:08 | |
| *** dave-mccowan has joined #openstack-containers | 13:11 | |
| *** dave-mcc_ has joined #openstack-containers | 13:20 | |
| *** dave-mccowan has quit IRC | 13:22 | |
| *** ricolin has joined #openstack-containers | 13:30 | |
| *** gsimondo1 has quit IRC | 13:59 | |
| *** ykarel is now known as ykarel|away | 14:04 | |
| *** mjura has quit IRC | 14:05 | |
| *** jmlowe has joined #openstack-containers | 14:12 | |
| *** ykarel|away has quit IRC | 14:12 | |
| *** mdnadeem has quit IRC | 14:15 | |
| *** sfilatov_ has joined #openstack-containers | 14:21 | |
| *** sfilatov has quit IRC | 14:21 | |
| *** armaan has quit IRC | 14:37 | |
| *** armaan has joined #openstack-containers | 14:38 | |
| *** jmlowe has quit IRC | 14:41 | |
| *** markguz has joined #openstack-containers | 14:43 | |
| *** janki has joined #openstack-containers | 14:45 | |
| *** markguz_ has joined #openstack-containers | 14:49 | |
| *** markguz has quit IRC | 14:52 | |
| *** vijaykc4 has quit IRC | 14:54 | |
| *** udesale has joined #openstack-containers | 14:56 | |
| *** hongbin has joined #openstack-containers | 15:00 | |
| *** rcernin_ has joined #openstack-containers | 15:05 | |
| *** hongbin has quit IRC | 15:08 | |
| *** hongbin has joined #openstack-containers | 15:08 | |
| *** udesale has quit IRC | 15:18 | |
| *** lpetrut has quit IRC | 15:19 | |
| *** ispp has quit IRC | 15:26 | |
| *** rcernin_ has quit IRC | 15:30 | |
| *** pcaruana has quit IRC | 15:32 | |
| *** lpetrut has joined #openstack-containers | 15:39 | |
| *** ispp has joined #openstack-containers | 15:41 | |
| *** itlinux has joined #openstack-containers | 15:49 | |
| *** lpetrut has quit IRC | 15:52 | |
| *** lpetrut has joined #openstack-containers | 16:03 | |
| *** ispp has quit IRC | 16:04 | |
| openstackgerrit | Spyros Trigazis proposed openstack/magnum master: WIP: build images in the ci https://review.openstack.org/585420 | 16:09 |
| *** serlex has quit IRC | 16:10 | |
| openstackgerrit | Andrei Ozerov proposed openstack/magnum master: Trustee: provide region_name to auth_url searching https://review.openstack.org/582947 | 16:22 |
| *** ricolin has quit IRC | 16:39 | |
| *** mjura has joined #openstack-containers | 16:39 | |
| *** markguz has joined #openstack-containers | 16:40 | |
| *** markguz_ has quit IRC | 16:43 | |
| *** vijaykc4 has joined #openstack-containers | 16:44 | |
| *** mgoddard has joined #openstack-containers | 17:06 | |
| *** itlinux_ has joined #openstack-containers | 17:07 | |
| *** itlinux has quit IRC | 17:09 | |
| *** AlexeyAbashkin has quit IRC | 17:11 | |
| *** lpetrut has quit IRC | 17:16 | |
| *** ramishra has quit IRC | 17:17 | |
| *** armaan has quit IRC | 17:25 | |
| canori01 | Just to clarify, so Tuesday meetings will move to 2200 UTC and no Thursday meetings? | 17:39 |
| *** itlinux_ has quit IRC | 17:42 | |
| *** itlinux has joined #openstack-containers | 17:42 | |
| *** vijaykc4 has quit IRC | 17:56 | |
| *** ykarel has joined #openstack-containers | 18:02 | |
| *** sfilatov_ has quit IRC | 18:02 | |
| *** zerick has quit IRC | 18:16 | |
| *** zerick_ has joined #openstack-containers | 18:16 | |
| *** ykarel is now known as ykarel|away | 18:33 | |
| *** zerick_ has quit IRC | 18:40 | |
| *** zerick has joined #openstack-containers | 18:41 | |
| *** itlinux has quit IRC | 18:54 | |
| *** mgoddard has quit IRC | 18:56 | |
| *** armaan has joined #openstack-containers | 18:57 | |
| *** itlinux has joined #openstack-containers | 19:02 | |
| *** itlinux has quit IRC | 19:03 | |
| *** ykarel|away is now known as mdnadeem | 19:03 | |
| *** mdnadeem has quit IRC | 19:03 | |
| *** mdnadeem has joined #openstack-containers | 19:03 | |
| *** mdnadeem has quit IRC | 19:08 | |
| *** canori01 has quit IRC | 19:15 | |
| *** armaan has quit IRC | 19:24 | |
| *** armaan has joined #openstack-containers | 19:25 | |
| *** armaan has quit IRC | 19:29 | |
| *** spiette has quit IRC | 19:40 | |
| *** spiette has joined #openstack-containers | 19:44 | |
| *** flwang1 has quit IRC | 19:55 | |
| *** rtjure has quit IRC | 19:57 | |
| *** itlinux has joined #openstack-containers | 19:59 | |
| *** canori01 has joined #openstack-containers | 20:06 | |
| imdigitaljim | stigazi: flwang1: will be at the #meetings tuesday :) | 20:14 |
| *** rcernin_ has joined #openstack-containers | 20:19 | |
| *** janki has quit IRC | 20:31 | |
| openstackgerrit | Spyros Trigazis proposed openstack/magnum master: WIP: build images in the ci https://review.openstack.org/585420 | 20:53 |
| strigazi | imdigitaljim flwang canori01 meeting? | 20:57 |
| *** flwang1 has joined #openstack-containers | 20:57 | |
| *** rtjure has joined #openstack-containers | 20:58 | |
| strigazi | team? | 21:01 |
| flwang | yes | 21:02 |
| flwang | i'm here | 21:02 |
| strigazi | imdigitaljim canori01 ? | 21:02 |
| strigazi | flwang: it doesn't make a lot of sense to make a meeting the two of us right? :) | 21:03 |
| strigazi | flwang: Was the email unclear? | 21:03 |
| flwang | yes, let's skip it then | 21:03 |
| flwang | probably because of the title of the mail | 21:04 |
| flwang | so not much people follow it? | 21:04 |
| strigazi | btw this seems to be working: https://review.openstack.org/585420 | 21:04 |
| *** jmlowe has joined #openstack-containers | 21:05 | |
| flwang | fantastic | 21:05 |
| flwang | but why the heat-container-agent image is being removed? | 21:06 |
| strigazi | it is not removed | 21:06 |
| strigazi | it is moved to /dockerfiles | 21:06 |
| strigazi | to have all containers under one dir | 21:07 |
| strigazi | It doesn't make sense to you? | 21:07 |
| strigazi | it was a shameless porting from openstack/loci | 21:08 |
| flwang | strigazi: ok, i'm reviewing it again | 21:08 |
| flwang | my fault, it's rename, not remove, my eyes ! | 21:09 |
| strigazi | the last patchset was to copy the logs | 21:09 |
| flwang | can we do the name convention then? | 21:10 |
| imdigitaljim | here | 21:10 |
| imdigitaljim | sorry | 21:10 |
| imdigitaljim | ive been working on a kubernetes cloud controller cleanup process using software deployment lifecycle hook | 21:11 |
| imdigitaljim | flwang1: great work with the heat-container-agent btw, i had been watching that and I appreciate the recent changes | 21:11 |
| flwang | imdigitaljim: thank you, that's a tricky because it needs a lot of collaboration from Heat and other teams | 21:12 |
| imdigitaljim | also we thought the first meeting was next week | 21:12 |
| imdigitaljim | our misunderstanding | 21:12 |
| imdigitaljim | strigazi: ^ | 21:13 |
| imdigitaljim | sorry i havent caught up some of my PR's, we've been pushing forward to get some operational and are going to upstream the changes after we can confirm much of it is indeed working | 21:14 |
| strigazi | imdigitaljim: no worries | 21:14 |
| imdigitaljim | in addition to some new features | 21:14 |
| strigazi | imdigitaljim: you have multiple regions too? | 21:14 |
| imdigitaljim | yeah | 21:14 |
| imdigitaljim | many regions | 21:15 |
| strigazi | one magnum to rule them all? | 21:15 |
| imdigitaljim | no a magnum in each region | 21:15 |
| imdigitaljim | although some components such as keystone cross multiple regions | 21:16 |
| flwang | imdigitaljim: same architecture at here | 21:16 |
| imdigitaljim | flwang1: good to know | 21:16 |
| strigazi | flwang: you will have many magnum deployments? | 21:17 |
| flwang | strigazi: yes | 21:17 |
| imdigitaljim | well for us technically HA magnum in each region | 21:18 |
| flwang | one for each region | 21:18 |
| strigazi | and many heats? | 21:18 |
| flwang | strigazi: yes | 21:18 |
| flwang | pls don't tell me bas story | 21:18 |
| flwang | bad | 21:18 |
| strigazi | flwang: no, I don't have any story | 21:19 |
| strigazi | :) | 21:19 |
| strigazi | btw I have a magnificent story | 21:19 |
| colin- | also here, sorry to be late | 21:19 |
| strigazi | about the issue with v1.11.x | 21:19 |
| strigazi | So 1.11.x was working perfectly in our production cloud but it wasn't working on devstack | 21:20 |
| imdigitaljim | i noticed 1.11.1 had some minor config changes needed to work | 21:21 |
| strigazi | the reason is that in 1.11 kubelet is proxing connections from the apiserver to the pods | 21:21 |
| imdigitaljim | were running on 1.11.0 right now though | 21:21 |
| flwang | strigazi: any reason we have to bump the rocky release to 1.11.1? | 21:22 |
| imdigitaljim | oh the error i had was related to invalid certificate key usages | 21:22 |
| strigazi | and kubelet was trying to server the container stream to localhost | 21:22 |
| strigazi | but the hyperkube container has the /etc/hosts file empty and clouldn't resolve localhost to 127.0.0.1 | 21:23 |
| flwang | strigazi: so you have figured out the root cause? | 21:23 |
| strigazi | well, at CERN we have out DNS and guess what, the dns resolves localhost.cern.ch to 127.0.0.1!!! | 21:24 |
| strigazi | hence it was working inprod | 21:24 |
| imdigitaljim | oh interesting | 21:24 |
| imdigitaljim | ill keep an eye out for that as well | 21:24 |
| strigazi | the solution was simple, mount /etc/hosts to the kubelet system container | 21:24 |
| imdigitaljim | we also didnt notice that DNS issue | 21:24 |
| flwang | imdigitaljim: that's only for 1.11.1 as you mentioned you're using 1.11.0 | 21:25 |
| strigazi | after that it worked | 21:25 |
| strigazi | flwang: in the end it was both | 21:25 |
| flwang | strigazi: will you propose a patch? | 21:25 |
| strigazi | yes | 21:25 |
| flwang | nice | 21:25 |
| imdigitaljim | flwang: no i mean i tried 1.11.1 briefly to see if we could just move to that without issue | 21:26 |
| strigazi | as soon as I fixed I cooked the builder to propose this patch in our repo | 21:26 |
| imdigitaljim | and it was failing based on some invalid keyusage | 21:26 |
| flwang | ok | 21:26 |
| imdigitaljim | i didnt spend much time on it after that but ill keep @strigazi: | 21:26 |
| imdigitaljim | in mind | 21:27 |
| strigazi | imdigitaljim the key usage comes from the above, double checking now. | 21:28 |
| strigazi | imdigitaljim: you can notice that the apiserver runs correctly but as soon as the kubelet tries to join the errors start to occur | 21:29 |
| imdigitaljim | strigazi: yeah i figured, i just didnt correlate that to /etc/hosts at the time, ill check it out as well | 21:29 |
| imdigitaljim | the info is much appreciated | 21:29 |
| *** rcernin_ has quit IRC | 21:30 | |
| strigazi | I need to check if we need /etc/hosts in all containers | 21:30 |
| imdigitaljim | did you say you were just adding them as a mount to config.json.template | 21:31 |
| strigazi | yes | 21:31 |
| imdigitaljim | perfect | 21:31 |
| imdigitaljim | ill validate my issue and give that a shot in a bit | 21:32 |
| flwang | great | 21:32 |
| flwang | btw, imdigitaljim as for the k8s cluster deployed by magnum, have you done any security audit? | 21:32 |
| imdigitaljim | we have no yet, we're waiting to have the resource cleanup fixed before we say we have an "MVP" | 21:33 |
| flwang | fair enough | 21:33 |
| flwang | let's share what we find then in that domain | 21:33 |
| imdigitaljim | absolutely | 21:34 |
| imdigitaljim | also strigazi: | 21:34 |
| flwang | it's very critical for prod using | 21:34 |
| imdigitaljim | minor thing that might help you out | 21:34 |
| imdigitaljim | in your Dockerfiles you could use | 21:34 |
| imdigitaljim | ARG KUBE_VERSION | 21:34 |
| imdigitaljim | FROM gcr.io/google-containers/kube-apiserver-amd64:$KUBE_VERSION | 21:34 |
| imdigitaljim | as the first 2 lines | 21:34 |
| imdigitaljim | and use --build-arg KUBE_VERSION={some param} | 21:35 |
| imdigitaljim | in the build to make updating a little easier | 21:35 |
| imdigitaljim | https://gitlab.cern.ch/cloud/atomic-system-containers/blob/cern-qa/kubernetes-apiserver/Dockerfile#L1 | 21:35 |
| imdigitaljim | so you might be able to do updates easier | 21:35 |
| imdigitaljim | if that works for your CI | 21:36 |
| *** itlinux has quit IRC | 21:38 | |
| strigazi | sounds good | 21:38 |
| strigazi | imdigitaljim: in the apiserver the error didn't went away yet | 21:39 |
| imdigitaljim | oh that invalid error i was talking about? | 21:39 |
| strigazi | imdigitaljim: flwang the kubelet error was fixed though | 21:39 |
| strigazi | imdigitaljim: yes | 21:39 |
| flwang | cool\ | 21:42 |
| strigazi | So, 1.11.0 is fixed. 1.11.1 needs some more work with rbac | 21:43 |
| imdigitaljim | yeah, thats where were at as well | 21:43 |
| imdigitaljim | 1.11.0 is good as is 1.11.1 needs some work | 21:44 |
| imdigitaljim | is it rbac or how we have our certs signed | 21:44 |
| flwang | 1.11.0 is good means no any work? | 21:44 |
| strigazi | in the CERN work though :( | 21:44 |
| strigazi | in the CERN cloud work though :( | 21:44 |
| flwang | 1.11.1 nees works, but have we figured out what we need to do? | 21:44 |
| strigazi | flwang: no, in 1.11.1 we miss something | 21:45 |
| strigazi | we don't know why | 21:45 |
| imdigitaljim | well if it works in the cern cloud still perhaps we can use it as a model for what is different =) | 21:45 |
| strigazi | in 1.11.0 we found the solution, we just need to push the images | 21:45 |
| strigazi | imdigitaljim: put localhost in the DNS server | 21:46 |
| imdigitaljim | yeah | 21:46 |
| strigazi | folks, I need to go, see you tmr | 21:47 |
| imdigitaljim | thanks for the discussion | 21:47 |
| imdigitaljim | take care | 21:47 |
| imdigitaljim | o/ | 21:47 |
| colin- | have a good one | 21:47 |
| strigazi | have a nice day all | 21:47 |
| strigazi | flwang: I might catch you later for you, tmr for me :) | 21:47 |
| strigazi | bye | 21:47 |
| cbrumm | strigazi: thanks for the meeting time change | 21:47 |
| strigazi | cbrumm: you are welcome | 21:48 |
| flwang | cy | 21:51 |
| openstackgerrit | Spyros Trigazis proposed openstack/magnum master: WIP: build images in the ci https://review.openstack.org/585420 | 21:58 |
| openstackgerrit | Spyros Trigazis proposed openstack/magnum master: WIP: build images in the ci https://review.openstack.org/585420 | 22:01 |
| *** jmlowe has quit IRC | 22:14 | |
| *** rcernin has joined #openstack-containers | 22:30 | |
| *** hongbin has quit IRC | 22:30 | |
| *** markguz has quit IRC | 23:10 | |
| *** yolanda_ has joined #openstack-containers | 23:56 | |
| *** linkmark has quit IRC | 23:57 | |
| *** yolanda has quit IRC | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
