| *** itlinux has quit IRC | 00:02 | |
| *** PagliaccisCloud has quit IRC | 00:09 | |
| *** dave-mccowan has joined #openstack-containers | 00:13 | |
| *** PagliaccisCloud has joined #openstack-containers | 00:15 | |
| *** dave-mccowan has quit IRC | 00:18 | |
| *** absubram has quit IRC | 00:38 | |
| openstackgerrit | Feilong Wang proposed openstack/magnum master: Support Keystone AuthN and AuthZ for k8s https://review.openstack.org/561783 | 00:42 |
|---|---|---|
| *** flwang has quit IRC | 01:05 | |
| openstackgerrit | Yang Le proposed openstack/magnum-ui master: Update mailing list from openstack-dev to openstack-discuss https://review.openstack.org/625744 | 01:12 |
| *** hongbin has joined #openstack-containers | 01:42 | |
| openstackgerrit | Feilong Wang proposed openstack/magnum master: Support Keystone AuthN and AuthZ for k8s https://review.openstack.org/561783 | 01:57 |
| openstackgerrit | Merged openstack/magnum master: [k8s] Cluster creation speedup https://review.openstack.org/623724 | 02:22 |
| *** hongbin_ has joined #openstack-containers | 02:28 | |
| *** hongbin has quit IRC | 02:29 | |
| *** ykarel|away has joined #openstack-containers | 02:46 | |
| *** ricolin has joined #openstack-containers | 02:48 | |
| openstackgerrit | Lingxian Kong proposed openstack/magnum stable/rocky: Add Octavia python client for Magnum https://review.openstack.org/625766 | 02:52 |
| openstackgerrit | Lingxian Kong proposed openstack/magnum stable/rocky: Delete Octavia loadbalancers for fedora atomic k8s driver https://review.openstack.org/625767 | 02:53 |
| openstackgerrit | Lingxian Kong proposed openstack/magnum stable/rocky: [k8s] Cluster creation speedup https://review.openstack.org/625768 | 02:54 |
| *** ykarel|away is now known as ykarel | 03:05 | |
| *** hongbin has joined #openstack-containers | 03:20 | |
| *** hongbin_ has quit IRC | 03:21 | |
| openstackgerrit | Feilong Wang proposed openstack/magnum master: Support Keystone AuthN and AuthZ for k8s https://review.openstack.org/561783 | 03:22 |
| *** rcernin has quit IRC | 03:25 | |
| *** rcernin has joined #openstack-containers | 03:27 | |
| *** rcernin has quit IRC | 03:28 | |
| *** rcernin has joined #openstack-containers | 03:28 | |
| openstackgerrit | Feilong Wang proposed openstack/magnum master: Support Keystone AuthN and AuthZ for k8s https://review.openstack.org/561783 | 03:30 |
| *** ramishra has joined #openstack-containers | 03:32 | |
| *** hongbin has quit IRC | 03:43 | |
| *** udesale has joined #openstack-containers | 04:11 | |
| *** ykarel is now known as ykarel|afk | 04:15 | |
| *** ykarel|afk has quit IRC | 04:19 | |
| *** Bhujay has joined #openstack-containers | 04:20 | |
| *** Bhujay has quit IRC | 04:21 | |
| *** Bhujay has joined #openstack-containers | 04:21 | |
| *** Bhujay has quit IRC | 04:22 | |
| *** Bhujay has joined #openstack-containers | 04:23 | |
| *** PagliaccisCloud has quit IRC | 04:31 | |
| *** PagliaccisCloud has joined #openstack-containers | 04:36 | |
| *** ykarel|afk has joined #openstack-containers | 04:39 | |
| *** ykarel|afk is now known as ykarel | 04:39 | |
| *** janki has joined #openstack-containers | 04:50 | |
| *** PagliaccisCloud has quit IRC | 05:01 | |
| *** itlinux has joined #openstack-containers | 05:45 | |
| openstackgerrit | Merged openstack/magnum-ui master: Update http link to https link https://review.openstack.org/623644 | 05:48 |
| *** PagliaccisCloud has joined #openstack-containers | 06:06 | |
| *** yolanda has joined #openstack-containers | 06:07 | |
| openstackgerrit | yfzhao proposed openstack/magnum master: Update mailinglist from dev to discuss https://review.openstack.org/625797 | 06:26 |
| *** PagliaccisCloud has quit IRC | 06:27 | |
| *** fragatina has joined #openstack-containers | 06:45 | |
| *** ykarel is now known as ykarel|lunch | 07:45 | |
| *** jonaspaulo_ has joined #openstack-containers | 07:51 | |
| *** ykarel|lunch is now known as ykarel | 08:51 | |
| openstackgerrit | Spyros Trigazis proposed openstack/magnum master: k8s_fedora: Use external kubernetes/cloud-provider-openstack https://review.openstack.org/577477 | 09:02 |
| strigazi | brtknr: ^^ | 09:05 |
| brtknr | I saw, just reading this: https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/openstack-kubernetes-integration-options.md | 09:06 |
| brtknr | strigazi: which agrees with your change | 09:06 |
| brtknr | strigazi: although "openstack.org/standalone-cinder" also appears to be supported | 09:07 |
| brtknr | strigazi: not sure how much longer "kubernetes.io/cinder" will be around | 09:08 |
| strigazi | brtknr: we should use csi-cinder but we can leave with kubernetes.io/cinder until we use csi. For catalyst the cloud provider is the most imporant feature afaik. At CERN we will mostly disable. CPO is still a moving target. | 09:16 |
| strigazi | s/leave/live/ | 09:16 |
| *** rcernin has quit IRC | 09:18 | |
| *** Bhujay has quit IRC | 09:24 | |
| brtknr | strigazi: just spinning up devstack to test the changes... especially standalone-cinder and why the file must be called cloud-config | 09:45 |
| *** fragatina has quit IRC | 09:48 | |
| *** _fragatina_ has joined #openstack-containers | 09:48 | |
| *** _fragatina_ has quit IRC | 09:48 | |
| *** fragatina has joined #openstack-containers | 09:48 | |
| strigazi | brtknr: dims said so :) http://eavesdrop.openstack.org/irclogs/%23openstack-containers/%23openstack-containers.2018-12-17.log.html#t2018-12-17T17:28:47 | 09:51 |
| brtknr | strigazi: *bangs head* argh | 09:53 |
| strigazi | brtknr: did you find the code? | 10:00 |
| strigazi | brtknr: did you find the line code? | 10:00 |
| *** Bhujay has joined #openstack-containers | 10:02 | |
| *** Bhujay has quit IRC | 10:03 | |
| *** Bhujay has joined #openstack-containers | 10:04 | |
| openstackgerrit | Feilong Wang proposed openstack/magnum master: Support Keystone AuthN and AuthZ for k8s https://review.openstack.org/561783 | 10:05 |
| *** Bhujay has quit IRC | 10:05 | |
| *** Bhujay has joined #openstack-containers | 10:05 | |
| *** Bhujay has quit IRC | 10:06 | |
| *** Bhujay has joined #openstack-containers | 10:07 | |
| openstackgerrit | Spyros Trigazis proposed openstack/magnum master: k8s_fedora: Use external kubernetes/cloud-provider-openstack https://review.openstack.org/577477 | 10:08 |
| mkuf | hi there, i'm trying to deploy a kubernetes cluster in queens. the master node gets deployed and cloudinit finishes and calls wc-notify sccessfully but no minions get deployed. any idea what might cause this behaviour? | 10:09 |
| *** flwang1 has joined #openstack-containers | 10:09 | |
| flwang1 | strigazi: did you test pvc? | 10:09 |
| *** ttsiouts has joined #openstack-containers | 10:10 | |
| strigazi | flwang1: this https://github.com/kubernetes/cloud-provider-openstack/blob/master/examples/persistent-volume-provisioning/cinder/cinder-in-tree-full.yaml | 10:12 |
| flwang1 | strigazi: great | 10:12 |
| flwang1 | is the PS9 ready for testing? | 10:13 |
| *** salmankhan has joined #openstack-containers | 10:16 | |
| strigazi | flwang1: and this https://github.com/kubernetes/cloud-provider-openstack/blob/master/examples/volumes/cinder/cinder-web.yaml | 10:16 |
| strigazi | flwang1: ^^ s/tcp/TCP/ | 10:16 |
| strigazi | flwang1: for cinder-in-tree-full.yaml the volume is created but not mounted. | 10:16 |
| flwang1 | strigazi: hmm... | 10:18 |
| flwang1 | we probably need a fix in CPO, lingxian mentioned this with me today | 10:19 |
| strigazi | Warning FailedScheduling 6s (x14 over 61s) default-scheduler 0/2 nodes are available: 1 node(s) had no available volume zone, 1 node(s) had taints that the pod didn't tolerate. | 10:21 |
| strigazi | probably this ^^ | 10:21 |
| strigazi | flwang1: 1 node(s) had no available volume zone | 10:21 |
| flwang1 | strigazi: got, i will double check with lxkong tomorrow | 10:22 |
| flwang1 | today i mainly working on the keystone patch and draino | 10:23 |
| strigazi | flwang1: without the problem detector? is this done? | 10:30 |
| flwang1 | without the node problem detector, it's still working by checking the condition of the node | 10:32 |
| flwang1 | based on my understanding, NPD just give you more space to define the detect policies, need more dig | 10:32 |
| flwang1 | but i'm pretty sure draino can work without NPD | 10:33 |
| lxkong | strigazi: to make pvc work, you need this https://github.com/kubernetes/cloud-provider-openstack/pull/405 | 10:34 |
| flwang1 | what draino can do is checking the condition of the node and then cordon the node, and after a while, drain it | 10:34 |
| strigazi | lxkong: flwang1 this is why we abandoned the cloud provider. | 10:35 |
| flwang1 | sorry? | 10:36 |
| flwang1 | strigazi: what have we abandoned? | 10:36 |
| lxkong | after that pr is merged and a few more role definition fixed, we are good to go | 10:36 |
| lxkong | i see in that cloud provider patch, just give the cloud-controller-manager a cluster-admin role which is not recommended | 10:37 |
| strigazi | we, at cern, stopped using teh cloud proider because of all these issues. | 10:37 |
| strigazi | lxkong: do you want to share what is recommended? | 10:38 |
| strigazi | lxkong: and push a patch here I guess https://kubernetes.io/docs/tasks/administer-cluster/running-cloud-controller/ | 10:39 |
| lxkong | https://raw.githubusercontent.com/kubernetes/cloud-provider-openstack/master/cluster/addons/rbac/cloud-controller-manager-roles.yaml | 10:39 |
| lxkong | and https://raw.githubusercontent.com/kubernetes/cloud-provider-openstack/master/cluster/addons/rbac/cloud-controller-manager-role-bindings.yaml | 10:39 |
| lxkong | i also have a fix for the role https://github.com/kubernetes/cloud-provider-openstack/pull/404 | 10:39 |
| strigazi | Is there any documentation for this decisions? :) | 10:40 |
| strigazi | Is there any documentation for these decisions? :) | 10:40 |
| *** udesale has quit IRC | 10:40 | |
| lxkong | what decision? | 10:40 |
| strigazi | Why the recommended roles/rolebindings of the openstack cloud provider are different than the cloud-controller-manager? | 10:42 |
| strigazi | lxkong: is there a doc saying it is not recommended to use cluster-admin for the cloud-provider? | 10:43 |
| strigazi | flwang1: lxkong what do you want to do? wait for lxkong's patch to be released and merge the CPO patch? | 10:45 |
| lxkong | i don't think there is an official definition for the role/rolebindings that a ccm should use. Using cluster-admin is totally ok, but that gives the service too broad permission. | 10:45 |
| flwang1 | strigazi: any other thing depend on this patch? | 10:46 |
| strigazi | apart from the time I wasted on CPO, I don't think so | 10:47 |
| flwang1 | strigazi: haha | 10:48 |
| flwang1 | i totally understand your angry | 10:48 |
| flwang1 | i don't mind merging it now, but i'd like to see comments, docs and release notes to highlight this | 10:49 |
| strigazi | fair enough, when the fix is out we can just change the version | 10:50 |
| strigazi | Well, no I think we should point the cpo release page. | 10:50 |
| strigazi | We can say that the deafult CPO is using this version -> for fearures/bugs people should read the CPO release notes, thoughts? | 10:51 |
| flwang1 | strigazi: can we just use latest CPO? until we figure out a stable one? | 10:51 |
| *** salmankhan has quit IRC | 10:52 | |
| strigazi | flwang1: no, I think the cpo tag should match the kube_tag. v1.11.x is compatible with v0.2.0 | 10:52 |
| strigazi | flwang1: and then users can pick the tag they want, no? | 10:53 |
| flwang1 | strigazi: again, we need a matrix ;) | 10:53 |
| flwang1 | to be clear, i agree the cpo version should be always consistent with the k8s version | 10:54 |
| strigazi | do we? CPO-k8s compatibility should be documented in magnum? | 10:55 |
| flwang1 | not only cpo-k8s, but also other services we're supporting | 10:56 |
| strigazi | flwang1: we have a compatibility matrix, CPO is not there. | 11:07 |
| flwang1 | ok, that's not a big problem at this moment | 11:07 |
| flwang1 | if you can add a comment and a release note, i'm happy to approve it | 11:08 |
| strigazi | flwang1: what comment? that dynamic volume provisioning doesn't work? | 11:15 |
| flwang1 | yep | 11:15 |
| flwang1 | we need a TODO or FIXME to highlight the PVC doesn't work | 11:16 |
| strigazi | out of curiosity you are going to maintain a fork of k8s? | 11:16 |
| flwang1 | strigazi: we're trying our best to avoid to do that | 11:16 |
| *** ttsiouts has quit IRC | 11:17 | |
| flwang1 | however, we need to build a ci/cd of k8s in case there is any bug Google doesn't care but Catalyst Cloud care | 11:17 |
| strigazi | flwang1: what kind of ci/cd ? | 11:18 |
| *** ttsiouts has joined #openstack-containers | 11:18 | |
| strigazi | flwang1: the people involved on CPO have openlab testing | 11:18 |
| flwang1 | like cherrypick a PR into our k8s repo, then trigger a build to build all the k8s images and run tests..... | 11:18 |
| strigazi | flwang1: everything should be tested there PVC is not since it was discovered yesterday, correct? | 11:18 |
| flwang1 | not for CPO, for k8s | 11:19 |
| strigazi | for the external IPs thing? | 11:20 |
| strigazi | flwang1: you carrt more patches? | 11:20 |
| strigazi | flwang1: you carry more patches? | 11:20 |
| flwang1 | strigazi: yep, e.g. the missing ip one | 11:21 |
| flwang1 | that one has been tagged into 1.11.6 | 11:21 |
| *** zufar has joined #openstack-containers | 11:21 | |
| flwang1 | so we won't carry any patch now | 11:21 |
| *** brtknr has quit IRC | 11:22 | |
| *** ttsiouts has quit IRC | 11:22 | |
| dims | strigazi : o/ | 11:24 |
| openstackgerrit | Spyros Trigazis proposed openstack/magnum master: k8s_build: Build kubernetes v1.11.6 containers https://review.openstack.org/625884 | 11:26 |
| strigazi | flwang1: ^^ | 11:26 |
| strigazi | dims: hi | 11:26 |
| flwang1 | strigazi: lovely, thanks | 11:27 |
| openstackgerrit | Spyros Trigazis proposed openstack/magnum master: k8s_fedora: Use external kubernetes/cloud-provider-openstack https://review.openstack.org/577477 | 11:30 |
| strigazi | flwang1: I think I have everything here ^^ | 11:30 |
| flwang1 | strigazi: thank you | 11:31 |
| flwang1 | strigazi: i can't test it now, probably tomorrow, if i test it OK, then I will approve it | 11:32 |
| flwang1 | it's 0:32 now | 11:32 |
| strigazi | flwang1: sounds good! | 11:33 |
| flwang1 | have to go off, see you tomorrow | 11:34 |
| strigazi | flwang1: Good Night! | 11:34 |
| *** jmlowe has quit IRC | 11:37 | |
| *** jmlowe has joined #openstack-containers | 11:38 | |
| mkuf | i'm trying to deploy a kubernetes cluster in queens. the master node gets deployed, cloudinit finishes and calls wc-notify sccessfully but no minions get deployed. any idea what might cause this behaviour? | 11:44 |
| *** Bhujay has quit IRC | 11:46 | |
| strigazi | mkuf: share in paste.openstack.org the output of openstack stack resource list -n2 <stack_id> | 11:47 |
| *** ttsiouts has joined #openstack-containers | 11:47 | |
| mkuf | strigazi: http://paste.openstack.org/show/737549/ | 11:51 |
| *** Bhujay has joined #openstack-containers | 11:52 | |
| strigazi | mkuf: in the master node, /var/log/cloud-init-output.log and journalctl -u heat-container-agent --no-pager | 11:53 |
| *** Bhujay has quit IRC | 11:53 | |
| *** Bhujay has joined #openstack-containers | 11:53 | |
| *** salmankhan has joined #openstack-containers | 11:54 | |
| *** Bhujay has quit IRC | 11:54 | |
| *** Bhujay has joined #openstack-containers | 11:55 | |
| *** Bhujay has quit IRC | 11:56 | |
| *** Bhujay has joined #openstack-containers | 11:56 | |
| *** Bhujay has quit IRC | 11:57 | |
| *** Bhujay has joined #openstack-containers | 11:58 | |
| *** Bhujay has quit IRC | 11:59 | |
| mkuf | strigazi: cloud-init-output.log: http://paste.openstack.org/show/737538/ and journalctl -u heat-container-agent --no-pager: http://paste.openstack.org/show/737550/ | 11:59 |
| *** Bhujay has joined #openstack-containers | 11:59 | |
| mkuf | strigazi: huh, looking at the journalctl output, it seems pretty obious why it's not working. it's using the internal endpoint for heat, instead of the public one. | 11:59 |
| *** Bhujay has quit IRC | 12:00 | |
| *** Bhujay has joined #openstack-containers | 12:01 | |
| strigazi | mkuf: +1 | 12:01 |
| *** PagliaccisCloud has joined #openstack-containers | 12:07 | |
| mkuf | strigazi: where does heat-container-agent get it's config from? is it possibly some misconfig in heat.conf? | 12:08 |
| *** zufar has quit IRC | 12:09 | |
| *** janki has quit IRC | 12:17 | |
| *** udesale has joined #openstack-containers | 12:27 | |
| *** salmankhan has quit IRC | 12:27 | |
| *** ramishra has quit IRC | 12:27 | |
| *** salmankhan has joined #openstack-containers | 12:28 | |
| strigazi | mkuf: in master do atomic containers list --no-trunc | 12:36 |
| strigazi | mkuf: for the heat agent you should have queens-stable | 12:36 |
| *** ramishra has joined #openstack-containers | 12:37 | |
| mkuf | strigazi: looks like rawhide(?) http://paste.openstack.org/show/737554/ | 12:44 |
| *** brtknr has joined #openstack-containers | 12:46 | |
| *** jonaspaulo_ has quit IRC | 12:56 | |
| *** brtknr has quit IRC | 13:00 | |
| *** brtknr has joined #openstack-containers | 13:01 | |
| *** brtknr has quit IRC | 13:12 | |
| strigazi | mkuf: you need queens-stable there was a bug in os-collect-config that we fized | 13:12 |
| strigazi | s/fized/fixed | 13:13 |
| *** brtknr has joined #openstack-containers | 13:13 | |
| strigazi | mkuf: it is in stable but we haven't released. I'll do a release https://github.com/openstack/magnum/blob/stable/queens/magnum/drivers/common/templates/kubernetes/fragments/start-container-agent.sh | 13:15 |
| openstackgerrit | Spyros Trigazis proposed openstack/magnum stable/queens: functional: retrieve cluster to get stack_id https://review.openstack.org/624732 | 13:20 |
| openstackgerrit | Spyros Trigazis proposed openstack/magnum stable/queens: functional: use vexxhost-specific nodes with nested virt https://review.openstack.org/624608 | 13:20 |
| openstackgerrit | Spyros Trigazis proposed openstack/magnum stable/queens: functional: add body for delete_namespaced_service in k8s https://review.openstack.org/625907 | 13:20 |
| openstackgerrit | Spyros Trigazis proposed openstack/magnum stable/queens: functional: use default admission_control_list values https://review.openstack.org/625908 | 13:20 |
| *** ramishra has quit IRC | 13:22 | |
| strigazi | mkuf: I'll try to do a release today or tmr for queens | 13:22 |
| *** ramishra has joined #openstack-containers | 13:22 | |
| *** brtknr has quit IRC | 13:25 | |
| *** brtknr has joined #openstack-containers | 13:25 | |
| *** brtknr has quit IRC | 13:29 | |
| *** brtknr has joined #openstack-containers | 13:29 | |
| dims | strigazi : left a detailed note in https://review.openstack.org/#/c/577477/8 ( cc lxkong ) | 13:37 |
| dims | lxkong : you can use the notes in ^^ to update the doc in cloud-provider-openstack and then we can cut a release | 13:37 |
| strigazi | dims: thank you, I really appreciate it | 13:43 |
| strigazi | dims: this diff [0] covers already what you described in your comment if you have time to check. | 13:46 |
| strigazi | [0] https://review.openstack.org/#/c/577477/7..10 | 13:46 |
| mkuf | strigazi: sweet, i'll keep an eye out for it. :) thanks | 13:47 |
| dims | strigazi : done | 14:13 |
| *** ykarel is now known as ykarel|away | 14:18 | |
| *** PagliaccisCloud has quit IRC | 14:20 | |
| *** brtknr has quit IRC | 14:27 | |
| *** brtknr has joined #openstack-containers | 14:27 | |
| *** ykarel|away has quit IRC | 14:36 | |
| *** ttsiouts has quit IRC | 14:45 | |
| *** ttsiouts has joined #openstack-containers | 14:46 | |
| *** ttsiouts has quit IRC | 14:50 | |
| *** Bhujay has quit IRC | 14:53 | |
| *** ramishra_ has joined #openstack-containers | 14:54 | |
| *** ykarel has joined #openstack-containers | 14:55 | |
| *** ramishra has quit IRC | 14:56 | |
| *** ttsiouts has joined #openstack-containers | 14:58 | |
| *** itlinux has quit IRC | 15:00 | |
| brtknr | strigazi: isnt that the kube-controller-manager complaining? | 15:02 |
| *** munimeha1 has joined #openstack-containers | 15:05 | |
| strigazi | brtknr: the kube-controller-manager complains in the logs and at the "same" time reports ok xD | 15:06 |
| strigazi | brtknr: isn't that amazing? | 15:07 |
| brtknr | strigazi: lol, i think there has been much confusion over nothing.... we are all mostly in agreement here! | 15:07 |
| brtknr | dims: i think we are trying to say the same thing re where --cloud-config arg is required... kube-controller-manager and cloud-controller-manager... others use --cloud-provider=external arg so dont require --cloud-config= arg | 15:08 |
| dims | cool +1 | 15:09 |
| brtknr | except kube-controller-manager needs it for cinder! | 15:09 |
| strigazi | dims: all good, thanks | 15:10 |
| dims | brtknr : just to be clear ... kubelet does use the file, just not from --cloud-config, but hard coded path :) | 15:10 |
| dims | i asked to remove --cloud-config from kubelet command line as it is not picked up from the command line but from the hard coded path | 15:11 |
| dims | (so if someone tries to change the path and will be surprised that it is not picked up from there) | 15:11 |
| dims | thats the other twist | 15:11 |
| brtknr | dims: oh really? how has it worked without the hard coded path all this time? since it has been /etc/kubernetes/kube_openstack_config... sorry im just trying to understand | 15:12 |
| brtknr | dims: for kubelet | 15:12 |
| *** hongbin has joined #openstack-containers | 15:14 | |
| dims | brtknr : because we are using the in-tree cloud provider before | 15:15 |
| dims | kubelet with --cloud-provider=openstack and --cloud-config=/etc/kubernetes/kube_openstack_config | 15:16 |
| brtknr | dims: ok, i thought --cloud-provider=external would be enough... | 15:19 |
| dims | brtknr : when we switch to csi or external provisioner, we won't need it. this clutch is because we need the "in-tree cinder provider" to still work | 15:21 |
| brtknr | ok, im cool, thanks :) i understand | 15:21 |
| brtknr | dims: how is the progress with external provisioner for cinder? | 15:22 |
| dims | brtknr : "Standalone-cinder external provisioner" works for rbd/iscsi based providers | 15:24 |
| *** ramishra_ has quit IRC | 15:27 | |
| brtknr | dims: i see, thanks. i imagine that includes softiron ceph based cinder? | 15:28 |
| dims | brtknr: someone has to try it ... dunno | 15:28 |
| brtknr | strigazi: hmm etcd is having trouble starting on my devstack checkout of the latest patch | 15:37 |
| strigazi | brtknr: which patch? | 15:38 |
| brtknr | strigazi: http://paste.openstack.org/show/737576/ | 15:39 |
| brtknr | https://review.openstack.org/#/c/577477/10 | 15:39 |
| strigazi | brtknr: some timeout in cloud-init-output.log I suppose | 15:40 |
| brtknr | Job for flanneld.service failed because a timeout was exceeded. | 15:41 |
| strigazi | higher in the scritp | 15:42 |
| brtknr | http://paste.openstack.org/show/737578/ | 15:44 |
| brtknr | Failed running /var/lib/cloud/instance/scripts/part-011 | 15:45 |
| strigazi | what is in ^^ | 15:46 |
| strigazi | ? | 15:46 |
| brtknr | essentially, my USER_TOKEN is coming back empty | 15:50 |
| brtknr | USER_TOKEN=`curl $VERIFY_CA -s -i -X POST -H "$content_type" -d "$auth_json" $url \ | 15:50 |
| brtknr | | grep -i X-Subject-Token | awk '{print $2}' | tr -d '[[:space:]]'` | 15:50 |
| brtknr | wait, i think i understand the problem | 15:54 |
| brtknr | strigazi: my private-subnet cidr is the same as my host | 15:55 |
| strigazi | so you can't reach keystone | 15:56 |
| strigazi | brtknr: correct? | 15:57 |
| brtknr | strigazi: yep | 15:57 |
| *** itlinux has joined #openstack-containers | 16:04 | |
| *** itlinux_ has joined #openstack-containers | 16:08 | |
| *** itlinux has quit IRC | 16:12 | |
| *** dave-mccowan has joined #openstack-containers | 16:13 | |
| brtknr | strigazi: are you using octavia for lbaas? | 16:17 |
| brtknr | or just neutron? | 16:17 |
| *** dave-mccowan has quit IRC | 16:18 | |
| strigazi | brtknr: at CERN nothing, something in-house with our DNS and haproxy. devstack -> octavia | 16:19 |
| *** dave-mccowan has joined #openstack-containers | 16:21 | |
| *** fragatina has quit IRC | 16:22 | |
| *** dave-mccowan has quit IRC | 16:22 | |
| *** fragatina has joined #openstack-containers | 16:24 | |
| *** dave-mccowan has joined #openstack-containers | 16:26 | |
| *** fragatina has quit IRC | 16:41 | |
| *** fragatina has joined #openstack-containers | 16:41 | |
| *** fragatina has quit IRC | 16:45 | |
| *** fragatina has joined #openstack-containers | 16:45 | |
| *** udesale has quit IRC | 16:49 | |
| *** sapd1 has joined #openstack-containers | 16:52 | |
| *** fragatina has quit IRC | 17:03 | |
| *** fragatina has joined #openstack-containers | 17:04 | |
| *** fragatina has quit IRC | 17:06 | |
| *** fragatina has joined #openstack-containers | 17:07 | |
| *** fragatina has quit IRC | 17:09 | |
| *** fragatina has joined #openstack-containers | 17:09 | |
| *** fragatina has quit IRC | 17:10 | |
| *** fragatina has joined #openstack-containers | 17:11 | |
| *** ttsiouts has quit IRC | 17:14 | |
| *** ttsiouts has joined #openstack-containers | 17:15 | |
| *** fragatina has quit IRC | 17:16 | |
| *** fragatina has joined #openstack-containers | 17:16 | |
| *** ricolin has quit IRC | 17:17 | |
| *** ttsiouts has quit IRC | 17:19 | |
| *** fragatina has quit IRC | 17:22 | |
| *** fragatina has joined #openstack-containers | 17:22 | |
| *** fragatina has quit IRC | 17:24 | |
| *** fragatina has joined #openstack-containers | 17:24 | |
| *** fragatina has quit IRC | 17:25 | |
| *** fragatina has joined #openstack-containers | 17:26 | |
| *** fragatina has quit IRC | 17:30 | |
| *** fragatina has joined #openstack-containers | 17:31 | |
| *** fragatina has quit IRC | 17:34 | |
| *** fragatina has joined #openstack-containers | 17:35 | |
| *** fragatina has quit IRC | 17:37 | |
| *** salmankhan has quit IRC | 17:37 | |
| *** fragatina has joined #openstack-containers | 17:37 | |
| *** fragatina has quit IRC | 17:39 | |
| *** fragatina has joined #openstack-containers | 17:40 | |
| *** fragatina has quit IRC | 17:45 | |
| *** fragatina has joined #openstack-containers | 17:46 | |
| *** fragatina has quit IRC | 17:47 | |
| *** fragatina has joined #openstack-containers | 17:47 | |
| *** jmlowe has quit IRC | 17:49 | |
| *** fragatina has quit IRC | 17:51 | |
| *** fragatina has joined #openstack-containers | 17:51 | |
| *** fragatina has quit IRC | 17:56 | |
| *** fragatina has joined #openstack-containers | 17:56 | |
| *** fragatina has quit IRC | 17:57 | |
| *** fragatina has joined #openstack-containers | 17:57 | |
| brtknr | Any reason this shouldnt be cherrypicked to queens? | 17:58 |
| brtknr | https://review.openstack.org/#/c/623724/14 | 17:58 |
| *** fragatina has quit IRC | 17:58 | |
| *** fragatina has joined #openstack-containers | 17:59 | |
| *** fragatina has quit IRC | 18:00 | |
| *** fragatina has joined #openstack-containers | 18:01 | |
| brtknr | strigazi: ^ | 18:01 |
| *** fragatina has quit IRC | 18:01 | |
| *** fragatina has joined #openstack-containers | 18:02 | |
| *** fragatina has quit IRC | 18:02 | |
| *** fragatina has joined #openstack-containers | 18:03 | |
| *** fragatina has quit IRC | 18:06 | |
| *** jmlowe has joined #openstack-containers | 18:06 | |
| *** fragatina has joined #openstack-containers | 18:06 | |
| *** sapd1 has quit IRC | 18:08 | |
| *** fragatina has quit IRC | 18:10 | |
| *** fragatina has joined #openstack-containers | 18:11 | |
| mnaser | hey all | 18:16 |
| *** PagliaccisCloud has joined #openstack-containers | 18:20 | |
| *** itlinux_ has quit IRC | 18:21 | |
| *** jmlowe has quit IRC | 18:21 | |
| *** itlinux has joined #openstack-containers | 18:21 | |
| *** jmlowe has joined #openstack-containers | 18:22 | |
| openstackgerrit | Mohammed Naser proposed openstack/magnum master: wip: multinode conformance https://review.openstack.org/625448 | 18:22 |
| *** ykarel has quit IRC | 18:24 | |
| *** jmlowe has quit IRC | 18:36 | |
| *** fragatina has quit IRC | 18:38 | |
| *** jmlowe has joined #openstack-containers | 18:43 | |
| *** jmlowe has quit IRC | 18:53 | |
| *** flwang1 has quit IRC | 18:58 | |
| *** jmlowe has joined #openstack-containers | 19:21 | |
| *** salmankhan has joined #openstack-containers | 19:34 | |
| *** salmankhan has quit IRC | 19:39 | |
| openstackgerrit | Mohammed Naser proposed openstack/magnum master: wip: multinode conformance https://review.openstack.org/625448 | 19:46 |
| *** jmlowe has quit IRC | 19:53 | |
| openstackgerrit | Mohammed Naser proposed openstack/magnum master: containers: clean-up build code https://review.openstack.org/625996 | 20:03 |
| *** jmlowe has joined #openstack-containers | 20:11 | |
| *** flwang has joined #openstack-containers | 20:41 | |
| *** fragatina has joined #openstack-containers | 20:49 | |
| flwang | strigazi: do we have meeting today? | 20:54 |
| strigazi | flwang: yes | 20:58 |
| flwang | cool | 20:58 |
| strigazi | #startmeeting containers | 21:00 |
| openstack | Meeting started Tue Dec 18 21:00:03 2018 UTC and is due to finish in 60 minutes. The chair is strigazi. Information about MeetBot at http://wiki.debian.org/MeetBot. | 21:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 21:00 |
| *** openstack changes topic to " (Meeting topic: containers)" | 21:00 | |
| openstack | The meeting name has been set to 'containers' | 21:00 |
| strigazi | #topic Roll Call | 21:00 |
| *** openstack changes topic to "Roll Call (Meeting topic: containers)" | 21:00 | |
| strigazi | o/ | 21:00 |
| cbrumm_ | o/ | 21:00 |
| schaney | o/ | 21:00 |
| eandersson | o/ | 21:01 |
| flwang | o/ | 21:02 |
| strigazi | #topic Announcements | 21:02 |
| *** openstack changes topic to "Announcements (Meeting topic: containers)" | 21:02 | |
| strigazi | I'd like to make a small announcement and thank mnaser, thanks to vexxhost we have a good ci for magnum after a long time. Take a look here: | 21:02 |
| strigazi | https://review.openstack.org/#/c/577477/ | 21:03 |
| mnaser | :D -- i hope to bring it to voting soon | 21:03 |
| mnaser | and im working here to get conformance tests passing in gates -- https://review.openstack.org/#/c/625448/ (ill let you finish the meeting and announcement =]) | 21:03 |
| cbrumm_ | nice | 21:03 |
| *** kosa777777 has joined #openstack-containers | 21:03 | |
| *** PagliaccisCloud has quit IRC | 21:04 | |
| strigazi | devstack and a k8s cluster ready in 48'. For the openstack ci i'm used to this is super fast, we can do even better I guess. | 21:04 |
| strigazi | thanks mnaser | 21:05 |
| strigazi | #topic Stories/Tasks | 21:05 |
| *** openstack changes topic to "Stories/Tasks (Meeting topic: containers)" | 21:05 | |
| strigazi | Last week I picked Jim's patch and finally made almost everything work: https://review.openstack.org/#/c/577477/ flwang and others you can review. | 21:06 |
| flwang | strigazi: i tested it yesterday before you add more comments | 21:07 |
| flwang | and it works except the known pvc issue | 21:07 |
| strigazi | Also last friday/saturday, I finished building all containers in the ci, here is a patch to build v1.11.6 https://review.openstack.org/#/c/625884/ | 21:08 |
| cbrumm_ | thumbs up | 21:09 |
| *** hongbin has quit IRC | 21:09 | |
| strigazi | finally from me, this branch is to make the ci in queens green https://review.openstack.org/#/q/status:open+project:openstack/magnum+branch:stable/queens+topic:624132 | 21:09 |
| *** hongbin has joined #openstack-containers | 21:09 | |
| strigazi | Maybe we miss one more, I'll have a look. | 21:09 |
| strigazi | That's it from me | 21:10 |
| flwang | on my side, i continually polished the keystone auth patch | 21:11 |
| flwang | and it's ready for another review | 21:12 |
| flwang | and help testing the CPO patch and the resource clean up patch | 21:12 |
| flwang | and I really love the speed up patch from mnaser and lxkong | 21:12 |
| strigazi | Looks ready to me, I'll test after the meeting for keystone auth. | 21:12 |
| mnaser | :D thanks for lxkong to polishing up my work | 21:13 |
| flwang | meanwhile, i'm working on the auto healing feature with NPD, draino, autoscaler | 21:13 |
| flwang | i have played draino a lot, and next will be NPD, then I will start to integrate those three | 21:13 |
| flwang | that's all | 21:13 |
| cbrumm_ | I'd love to hear your thoughts on those flwang | 21:14 |
| schaney | We are also starting to look into those, hopefully we'll have more to discuss in starting in the new year | 21:14 |
| flwang | cbrumm_: we can discuss offline | 21:14 |
| flwang | schaney: sure thing | 21:15 |
| openstackgerrit | Mohammed Naser proposed openstack/magnum master: wip: multinode conformance https://review.openstack.org/625448 | 21:16 |
| *** jakeyip has joined #openstack-containers | 21:16 | |
| strigazi | Does anyone have something else to bring up? | 21:18 |
| cbrumm_ | None hear, we're deploying magnum to production but we won't be opening it up for use until later. Right now we're just trying to get to the new year. | 21:19 |
| jakeyip | hi all sorry am late | 21:19 |
| strigazi | cbrumm_: +! | 21:19 |
| flwang | cbrumm_: mind letting us know your company? | 21:19 |
| strigazi | cbrumm_: +1 | 21:19 |
| cbrumm_ | Blizzard | 21:20 |
| flwang | oh, no | 21:20 |
| flwang | i think you already got Magnum on prod, no? | 21:20 |
| cbrumm_ | Not prod, been in dev for months | 21:20 |
| colin- | also here lurking, nothing to introduce today | 21:20 |
| flwang | ah, i see. sorry, i thought you're in Blizzard | 21:20 |
| colin- | we are (including cbrumm) | 21:20 |
| schaney | I as well | 21:21 |
| flwang | colin-: yep, i know | 21:21 |
| eandersson | o/ | 21:21 |
| flwang | hah | 21:21 |
| flwang | LOL | 21:21 |
| flwang | jakeyip: anything you want to discuss? | 21:21 |
| jakeyip | wow that's a bunch of people. hi Blizzard | 21:21 |
| jakeyip | not from me, lurking in the background | 21:22 |
| colin- | testing out 1.13.1 and trying to get ipvs happy with the change they /q flwang | 21:22 |
| colin- | fixed in the patch, that i sent you in PM flwang (sorry for typo) | 21:22 |
| strigazi | colin-: is it in gerrit? | 21:22 |
| *** rcernin has joined #openstack-containers | 21:22 | |
| colin- | no, built from upstream to see if it was likely to introduce any issues with the control plane. nothing distressing yet but did see a heat-agent trace i will need to investigate | 21:23 |
| flwang | colin-: great | 21:23 |
| strigazi | for k8s 1.13.1? | 21:23 |
| colin- | yes | 21:24 |
| strigazi | Shall we go to 1.13.x? it works as far as I'm concerned and conformance tests are passing. | 21:24 |
| colin- | i'm in favor of it, personally, given the CVE | 21:25 |
| strigazi | tested in magnum master devstack and in prod at cern. | 21:25 |
| colin- | did you have to change anything? | 21:25 |
| strigazi | colin-: 1.11.5 covers the CVE | 21:25 |
| colin- | that's fair, just eager for the 1.13 goodies i suppose | 21:26 |
| strigazi | colin-: the scheduler tries to generate certs. I forced to listen in only yo localhost and serve insecurely | 21:26 |
| strigazi | colin-: the scheduler tries to generate certs. I forced to listen only on localhost and serve insecurely | 21:26 |
| strigazi | Regarding, the next meeting, I'll manage to be around only on the 9th of January. I won't manage for the next two tuesdays. | 21:28 |
| cbrumm_ | Same for us | 21:28 |
| strigazi | flwang? next meeting on the 9th of January 2019? | 21:29 |
| colin- | i'll take a look at the scheduler startup for that behavior, thanks strigazi | 21:29 |
| flwang | strigazi: 9th works for me | 21:29 |
| flwang | i will take leave for next 2 weeks | 21:29 |
| jakeyip | hee some here :) | 21:30 |
| flwang | jakeyip: celebrate the summer Xmas :D | 21:30 |
| jakeyip | going to be hot hot hot | 21:30 |
| jakeyip | before everyone leaves I have something not terriby urgent that I would like to ask | 21:31 |
| strigazi | jakeyip: go for it | 21:31 |
| jakeyip | I've been talking to flwang about a use case having floating IPs only for masters so that we can save some v4 addresses. from what we've found it doesn't work. it's not terrribly urgent but wondering if this is something magnum is willing to support? | 21:32 |
| colin- | interesting, what is the objective for it? | 21:32 |
| strigazi | jakeyip: yes, it is. We lack manpower at the moment though. | 21:33 |
| flwang | strigazi: i think you have a patch for that? | 21:33 |
| strigazi | colin-: I think, usally fips == expensive public ipv4 | 21:33 |
| flwang | or somebody else, but im pretty sure there is a patch somewhere | 21:33 |
| jakeyip | colin-: well we would rather have a cluster that only uses private network addresses internally, but has FIP so users can use kubectl from outside | 21:34 |
| strigazi | flwang: yes, I do, needs more work though, not just a rebase. | 21:34 |
| flwang | strigazi: i see and i think the hard part is the backward compatibility | 21:34 |
| jakeyip | strigazi: can point me to it? I don't mind having a look and see what I can do | 21:34 |
| flwang | jakeyip: if you have bandwidth, please just take it and we're happy to review | 21:35 |
| strigazi | https://review.openstack.org/#/c/395095/ | 21:35 |
| jakeyip | thanks I'll take a look and see what can be done | 21:35 |
| strigazi | cool | 21:36 |
| colin- | understood | 21:37 |
| *** rcernin has quit IRC | 21:37 | |
| *** fragatina has quit IRC | 21:37 | |
| strigazi | Shall we end the meeting? Anything else to discuss? | 21:38 |
| cbrumm_ | I think we're good | 21:39 |
| strigazi | cool, thanks everyone. See you next year! Happy holidays | 21:41 |
| jakeyip | happy holidays! | 21:41 |
| strigazi | #endmeeting | 21:42 |
| *** openstack changes topic to "OpenStack Containers Team" | 21:42 | |
| openstack | Meeting ended Tue Dec 18 21:42:03 2018 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 21:42 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/containers/2018/containers.2018-12-18-21.00.html | 21:42 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/containers/2018/containers.2018-12-18-21.00.txt | 21:42 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/containers/2018/containers.2018-12-18-21.00.log.html | 21:42 |
| flwang | strigazi: thank you! | 21:42 |
| flwang | cbrumm_: let me know if you want to discuss auto healing | 21:42 |
| cbrumm_ | Ok flwang, will need to be later though. The rest of my day is pretty busy. | 21:44 |
| flwang | cbrumm_: no problem | 21:45 |
| flwang | we just start to investigate, so just ping me when you want to discuss | 21:45 |
| flwang | strigazi: and cern are also working on that | 21:45 |
| cbrumm_ | that's like us, we're just installing it to see if we want to extend it. Big question is what do we want to test for? | 21:46 |
| flwang | automatically replace a bad node | 21:46 |
| cbrumm_ | sure, but what makes a node bad? How do you detect that it's a node you want to kill? | 21:47 |
| cbrumm_ | out of the box NPD looks like it only checks for kernel error messages. I think I'm more concerned with things like node offline, docker daemon stuck, things like that | 21:48 |
| jakeyip | sorry for interrupting - is this autohealing on magnum / heat side? e.g. bring up a new VM if you kill the old one? | 21:49 |
| flwang | jakeyip: on top of k8s | 21:50 |
| cbrumm_ | heat autoscaling, so long as we remove bad vms the autoscaling will replace it. | 21:50 |
| flwang | not magnum/heat, but autoscaler will call magnum/heat to do node replace | 21:50 |
| flwang | cbrumm_: that's a very good question | 21:50 |
| flwang | in my testing, NPD just detect some error and send an event | 21:51 |
| flwang | but draino will directly check the condition of a node and then cordon/drain it | 21:51 |
| flwang | so technically, for simple case, draino can work without NPD | 21:51 |
| cbrumm_ | true, don't need both | 21:51 |
| flwang | after the node drained, autoscaler can replace it by calling magnum api (may need new api endpint) | 21:52 |
| flwang | and for NPD, i think we probably need to extend it if we want to cover more scenarios | 21:52 |
| cbrumm_ | exactly | 21:52 |
| cbrumm_ | that's what I'm interested in, a set of tests that we can all share and contribute to | 21:53 |
| cbrumm_ | right now we're not even sure what scenarios we want to test for | 21:53 |
| cbrumm_ | that's work we have planned for Jan | 21:54 |
| flwang | https://kubernetes.io/docs/tasks/debug-application-cluster/monitor-node-health/#kernel-monitor | 21:54 |
| flwang | correct, we need to figure out a minimum set of conditions we (magnum) care about | 21:55 |
| flwang | and then configure draino to monitor it | 21:55 |
| cbrumm_ | Yep, exactly | 21:55 |
| flwang | https://github.com/kubernetes/node-problem-detector/blob/v0.1/config/kernel-monitor.json | 21:56 |
| cbrumm_ | I'm thinking we might need other problem daemons besides KernelMonitor | 21:59 |
| cbrumm_ | something besides a log scraper | 21:59 |
| cbrumm_ | I have to go, we'll chat about this more later | 22:00 |
| flwang | sure, ttyl | 22:00 |
| cbrumm_ | bye all | 22:00 |
| mnaser | cbrumm_: awesome re blizzard | 22:00 |
| mnaser | can we know WHAT runs on magnum? :D | 22:00 |
| mnaser | strigazi, flwang: i've been thinking of running conformance tests or functional tests across different versions in master | 22:01 |
| mnaser | that way our users can always deploy latest 1.11, 1.12, 1.13 (and we know it works) | 22:01 |
| mnaser | and we also can add something to test master (but non voting) | 22:01 |
| flwang | mnaser: i like the idea | 22:02 |
| flwang | for example, we're running on 1.11.x which has been certificated, and we don't want to move very fast with the k8s version | 22:02 |
| flwang | so if we can have a gate job for those quite popular k8s versions, that would be fantastic | 22:03 |
| *** rcernin has joined #openstack-containers | 22:04 | |
| *** munimeha1 has quit IRC | 22:05 | |
| colin- | sure mnaser! many docker containers :D | 22:10 |
| mnaser | colin-: oh pfft | 22:10 |
| colin- | sorry, too easy | 22:10 |
| mnaser | i was hoping the next time i'd log in to bnet | 22:11 |
| mnaser | i'm launching into magnum-space | 22:11 |
| mnaser | ...or maybe when i open my phone.. ha. | 22:11 |
| mnaser | flwang: yep! that's my goal once i get conformance going | 22:11 |
| openstackgerrit | Mohammed Naser proposed openstack/magnum master: wip: multinode conformance https://review.openstack.org/625448 | 22:12 |
| flwang | mnaser: cool | 22:14 |
| mnaser | i think i'm close to getting the confromance tests to run on every commit | 22:16 |
| mnaser | which is *amazing* D: | 22:16 |
| openstackgerrit | Mohammed Naser proposed openstack/magnum master: containers: clean-up build code https://review.openstack.org/625996 | 22:35 |
| *** rcernin has quit IRC | 22:37 | |
| *** rcernin has joined #openstack-containers | 22:41 | |
| *** rcernin has quit IRC | 22:43 | |
| openstackgerrit | Mohammed Naser proposed openstack/magnum master: wip: multinode conformance https://review.openstack.org/625448 | 22:43 |
| *** rcernin has joined #openstack-containers | 22:45 | |
| *** itlinux has quit IRC | 22:56 | |
| *** yolanda has quit IRC | 23:55 | |
| *** dave-mccowan has quit IRC | 23:56 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!