| *** yingjun has joined #openstack-kolla | 00:07 | |
| *** manheim has quit IRC | 00:26 | |
| *** manheim has joined #openstack-kolla | 00:28 | |
| *** jtriley has joined #openstack-kolla | 00:29 | |
| sdake | rwellum are ou about? | 00:30 |
|---|---|---|
| sdake | rwellum i've sorted out the CNI stuff | 00:31 |
| *** adil452100 has joined #openstack-kolla | 00:31 | |
| sdake | rwellum and have a cleanup script that seems to work pretty well | 00:31 |
| sdake | sup spiette | 00:31 |
| sdake | sup spsurya | 00:31 |
| adil452100 | Hello again .. I want to ask some basic questions please | 00:36 |
| adil452100 | The OpenStack deployed via Kolla can be | 00:37 |
| adil452100 | Highly available not only | 00:38 |
| adil452100 | with controller | 00:38 |
| adil452100 | but also compute nodes ? | 00:38 |
| adil452100 | I mean if we loose a compute node, the VM started on that host will automagically migrate to another host ? (of course using ceph as a shared storage) | 00:39 |
| SamYaple | adil452100: there is no built-in mechanism to do that | 00:47 |
| SamYaple | adil452100: but there is a nova command to do that. host-evacuate | 00:47 |
| SamYaple | adil452100: it wont happen automatically | 00:47 |
| adil452100 | <SamYaple> No change to automate things on nova.conf ? | 00:48 |
| SamYaple | adil452100: there is no mechanism in openstack to automate things like that. you ahve to write it yourself | 00:48 |
| SamYaple | there are monitoring scripts and projects around that claim to do that. never seen one that worked well personally | 00:49 |
| SamYaple | but it might exist | 00:49 |
| adil452100 | So we can say for the time being, there is no native NOVA for a highly available VMs ? | 00:51 |
| SamYaple | i mean you can make your vms highly available by having multiple ones in different AZs | 00:53 |
| SamYaple | but there is no native mechanism to automatically migrate vms upon a host failure | 00:53 |
| SamYaple | there likely never will be | 00:53 |
| adil452100 | <SamYaple> Unfortunately I thought I could have something like VMware HA, but I will look for a work around with host-evacuate and share it if I have a good results | 00:55 |
| *** lucasxu has joined #openstack-kolla | 00:56 | |
| sdake | sbezverk all db creates are doing a crashloopbackoff when i launch the services independently as outlined in our deployment documentation - any idea how to fix that? | 00:56 |
| sdake | sbezverk i'm not sure if computekit works in this scenario - so it could be a bad cloud.yaml | 00:56 |
| *** lucasxu has quit IRC | 00:57 | |
| sdake | sbezverk i have verified that my envrionment works properly | 00:57 |
| sdake | sbezverk e.g.: cinder-create-db-nrkf4 0/2 CrashLoopBackOff | 00:57 |
| SamYaple | adil452100: right. host-evacuate can provide that funcationality, but youll need to be doing your own host monitoring and triggering the evac | 00:58 |
| SamYaple | adil452100: at the end of the day, VMWare is far superior in this regard, jsut fyi. dont expect openstack to compete in that area | 00:59 |
| sdake | adil452100 i suggest you try #openstack-nova with questions related to their plans for automated host evacuation, they may be changing | 01:00 |
| sdake | adil452100 kolla isn't responsible for nova's implementation - the nova team is | 01:00 |
| sdake | adil452100 the best we can do is speculate based upon past behavior, which has been anti-auto-evacation, however, the more they hear it as a requirement hte more likely the yhare to implement it | 01:01 |
| *** yangyapeng has joined #openstack-kolla | 01:01 | |
| sdake | adil452100 however for the time being there is no native nova HA vm technology (your original question) - there may be in the future if enough people ask for it | 01:03 |
| SamYaple | mind you they contantly reject this and have stated they wont do it as a policy. mainly because there is no safe way to do it for nova. nova simply doesnt have enough info to do it. (nova-compute doesnt have to be running for the vm to be working, and thats nova's only entry point) | 01:03 |
| sdake | SamYaple things change :) | 01:04 |
| adil452100 | <SamYaple> <sdake> Ok, I'll discuss it with #openstack-nova, but as I said earlier it's a basic question and I thought I could configure this functionality with Kolla. Anyway :) thanks for your answers and help. | 01:04 |
| *** yuanying_ has joined #openstack-kolla | 01:04 | |
| sdake | adil452100 the functionality can't be configured with kolla because nova doesn't implement it | 01:05 |
| *** iceyao has joined #openstack-kolla | 01:06 | |
| *** cuongnv has joined #openstack-kolla | 01:06 | |
| *** yuanying has quit IRC | 01:07 | |
| *** eaguilar has joined #openstack-kolla | 01:08 | |
| *** manheim has quit IRC | 01:08 | |
| adil452100 | Another question: Can Kolla use a disk partition for a ceph journal instead of a whole ssd disk ? | 01:09 |
| SamYaple | adil452100: yea | 01:10 |
| SamYaple | at least it did a while ago when i added that. i havent looked at it for a while | 01:11 |
| SamYaple | i know there was some trouble with partition namelegth | 01:11 |
| adil452100 | And a last question about upgrade, If I install now Newton, can I upgrade without downtime (VMs especially) to the next version Ocata ? The process is just to stop the old containers and to start the new ones or it needs a very complicated procedure to upgrade ? | 01:13 |
| *** iceyao has quit IRC | 01:14 | |
| *** iceyao has joined #openstack-kolla | 01:15 | |
| *** iceyao has quit IRC | 01:16 | |
| SamYaple | adil452100: its nova, assume the later | 01:16 |
| kolla-slack | <vhosakot> @sdake did you figure out dns_name in cloud.yaml. I see gate uses dns_name=172.18.0.1 which is not the IP address of any interface of the host. | 01:16 |
| kolla-slack | <vhosakot> https://github.com/openstack/kolla-kubernetes/blob/master/tests/bin/common_workflow_config.sh#L10 | 01:16 |
| kolla-slack | <vhosakot> https://github.com/openstack/kolla-kubernetes/blob/master/tools/setup_gate_ceph.sh#L9 | 01:16 |
| kolla-slack | <vhosakot> http://logs.openstack.org/03/449803/16/check/gate-kolla-kubernetes-deploy-ubuntu-binary-2-ceph-nv/b6e6230/console.html#_2017-03-26_19_24_49_845781 | 01:16 |
| SamYaple | adil452100: afaik there is still not a nova approved 0 downtime way to do things. there is always some downtime | 01:17 |
| kolla-slack | <vhosakot> @sdake see these 3 links about dns_name in gate ^^^ | 01:17 |
| adil452100 | <SamYaple> thx again for all the information .. that's kind of you :) | 01:20 |
| *** iceyao has joined #openstack-kolla | 01:20 | |
| *** jtriley has quit IRC | 01:20 | |
| *** iceyao has quit IRC | 01:21 | |
| adil452100 | <SamYaple> The upgarde can be done with Kolla even with a little downtime ? | 01:21 |
| *** caowei has joined #openstack-kolla | 01:22 | |
| *** zhubingbing has joined #openstack-kolla | 01:22 | |
| adil452100 | The Kolla process is just to stop the old containers and to start the new ones or it needs a very complicated procedure to upgrade ? | 01:24 |
| SamYaple | adil452100: its all handled in the ansible playbooks, but it can get complicated with the restart order | 01:25 |
| SamYaple | adil452100: its simple for _you_ | 01:25 |
| sbezverk | SamYaple: do you have a second for qq? | 01:27 |
| SamYaple | sbezverk: sure | 01:30 |
| *** daidv has joined #openstack-kolla | 01:30 | |
| sbezverk | SamYaple: I needed "customer" libvirtd config, but it seems merging custome config with default config is not working for libvirtd container | 01:31 |
| sbezverk | is it known limitation or I am doing something wrong? | 01:31 |
| sbezverk | I meant "custom" config | 01:32 |
| *** iceyao has joined #openstack-kolla | 01:32 | |
| SamYaple | sbezverk: afaik libvirtd isnt setup to be augmented | 01:32 |
| SamYaple | at all | 01:32 |
| sbezverk | SamYaple: thanks :( | 01:33 |
| SamYaple | sbezverk: https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/nova/tasks/config.yml#L79 | 01:33 |
| SamYaple | yea its not | 01:33 |
| sbezverk | SamYaple: ok, I guess I will have to run time modification of libvirtd.conf before the container starts.. | 01:34 |
| sbezverk | thanks for confirming it | 01:34 |
| *** eaguilar has quit IRC | 01:36 | |
| *** duonghq has joined #openstack-kolla | 01:40 | |
| *** tovin07 has joined #openstack-kolla | 01:41 | |
| *** adil452100 has quit IRC | 01:41 | |
| duonghq | moring guys | 01:42 |
| *** Pavo has joined #openstack-kolla | 01:46 | |
| *** jtriley has joined #openstack-kolla | 01:48 | |
| *** jtriley has quit IRC | 01:52 | |
| *** sbezverk has quit IRC | 01:53 | |
| *** dixiaoli has joined #openstack-kolla | 01:53 | |
| *** jtriley has joined #openstack-kolla | 01:54 | |
| zhubingbing | moring | 01:59 |
| *** wirehead_ has quit IRC | 02:01 | |
| *** jtriley has quit IRC | 02:02 | |
| *** wirehead_ has joined #openstack-kolla | 02:03 | |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla master: make yum fail on missing packages https://review.openstack.org/449086 | 02:05 |
| *** dave-mccowan has quit IRC | 02:09 | |
| *** eaguilar has joined #openstack-kolla | 02:09 | |
| *** dixiaoli has quit IRC | 02:21 | |
| *** yuanying has joined #openstack-kolla | 02:30 | |
| *** yuanying_ has quit IRC | 02:33 | |
| *** yangyapeng has quit IRC | 02:38 | |
| *** dixiaoli has joined #openstack-kolla | 02:39 | |
| *** dixiaoli has quit IRC | 03:00 | |
| *** dixiaoli has joined #openstack-kolla | 03:03 | |
| openstackgerrit | zhubingbing proposed openstack/kolla-kubernetes master: Fix problem with kollakube res create configmap https://review.openstack.org/439740 | 03:09 |
| *** caowei has quit IRC | 03:12 | |
| *** caowei has joined #openstack-kolla | 03:12 | |
| spsurya | duonghq: sup | 03:16 |
| duonghq | hi spsurya | 03:16 |
| spsurya | how was the weekend ? | 03:16 |
| *** caoyuan has joined #openstack-kolla | 03:24 | |
| duonghq | busy weekend :( | 03:24 |
| zhubingbing | ；） | 03:25 |
| masber | morning, I installed kolla-ansible and my router namespace can't ping the internet. I can see that kolla puts quantum gateway port in br-int, is that the right way? | 03:26 |
| masber | *port br-int | 03:26 |
| spsurya | same here | 03:29 |
| *** fooliouno has joined #openstack-kolla | 03:31 | |
| *** yangyapeng has joined #openstack-kolla | 03:33 | |
| *** lamt has joined #openstack-kolla | 03:34 | |
| sdake | vhosakot yes I sorted it out - its in the review for the guide | 03:40 |
| sdake | sup duonghq | 03:41 |
| sdake | sup zhubingbing | 03:41 |
| zhubingbing | hi sdake | 03:41 |
| duonghq | not sure why https://review.openstack.org/#/c/449086/ keep fail at gate-kolla-dsvm-build-centos-source-centos-7 test | 03:42 |
| duonghq | http://logs.openstack.org/86/449086/3/check/gate-kolla-dsvm-build-centos-source-centos-7/ac8a7dc/logs/build/keystone-base.txt.gz | 03:42 |
| duonghq | something wrong with packagecloud.io | 03:42 |
| duonghq | https://packagecloud.io/grafana/stable/el/7/x86_64/repodata/repomd.xml: [Errno 14] HTTPS Error 302 - Found | 03:42 |
| duonghq | (grafana repo) | 03:42 |
| zhubingbing | yeah | 03:43 |
| zhubingbing | grafana have some problems | 03:43 |
| duonghq | zhubingbing, did you notice that? | 03:43 |
| duonghq | it blocked gate | 03:43 |
| zhubingbing | I met a few days ago, did not solve | 03:43 |
| zhubingbing | centos source | 03:44 |
| zhubingbing | i fix it ；） | 03:44 |
| zhubingbing | wai | 03:44 |
| zhubingbing | wait | 03:44 |
| duonghq | zhubingbing, you have idea? | 03:45 |
| zhubingbing | yeah | 03:46 |
| zhubingbing | grafana.repo have some problems | 03:46 |
| sdake | should I b eable to nslookup kbuernetes from inside the busybox container portdirect ? | 03:47 |
| zhubingbing | yeah | 03:48 |
| zhubingbing | i think so sdake | 03:48 |
| zhubingbing | [root@kube1 ~]# cat /etc/resolv.conf | 03:50 |
| zhubingbing | # Generated by NetworkManager | 03:50 |
| zhubingbing | search kolla.svc.cluster.local svc.cluster.local cluster.local | 03:50 |
| zhubingbing | nameserver 172.16.35.10 | 03:50 |
| zhubingbing | options ndots:5 | 03:50 |
| zhubingbing | nameserver 68.105.28.11 | 03:50 |
| zhubingbing | nameserver 68.105.29.11 | 03:50 |
| zhubingbing | nameserver 68.105.28.12 | 03:50 |
| zhubingbing | sdake | 03:50 |
| zhubingbing | it is my host' resolve.conf | 03:50 |
| *** sbezverk has joined #openstack-kolla | 03:50 | |
| sdake | ok so nameserver is 172.16.35.10 - what is 172.16.35/8? | 03:50 |
| sdake | where is that set zhubingbing | 03:50 |
| sdake | here is what sbezverk said re the variables: | 03:51 |
| zhubingbing | yeah | 03:52 |
| zhubingbing | kubeadm init --skip-preflight-checks --use-kubernetes-version v1.5.4 --service-cidr 172.16.35.0/24 | 03:52 |
| *** dixiaoli has quit IRC | 03:52 | |
| sdake | [06:47:24] <rwellum>As a minimum, 'external_vip': your external ip address, 'ext_interface_name', 'ext_bridge_name' and 'tunnel_interface' - is what I had to change | 03:52 |
| sdake | what are these variables? | 03:53 |
| sdake | [06:50:09] <sbezverk_>external_vip: ip address assigned to your host and kubernetes api server listen on | 03:53 |
| sdake | [06:53:09] <sbezverk_>ext_interface_name: name of interface or bridge which will be used by neutron's provider interface | 03:53 |
| sdake | [06:54:16] <sbezverk_>ext_bridge_name: name of bridge you want neutron to use as external bridge, by default it shold be br-ex, but you can change it by using this option | 03:53 |
| sdake | [06:55:39] <sbezverk_>tunnel_interface: interface name used for connectivity between nodes in kubernetes cluster, in most of cases it matches the name you kubernetes host management interface | 03:53 |
| sdake | so service cidr | 03:53 |
| sdake | let me try that zhubingbing | 03:53 |
| zhubingbing | it is external_vip | 03:53 |
| zhubingbing | because we don't use keep live d | 03:53 |
| sdake | this is what I use: (reverse-i-search)`kubeadm': sudo kubeadm init --pod-network-cidr=10.1.0.0/16 --service-cidr=10.3.0.0/16 | 03:54 |
| sdake | so I will set my dns_name to 10.3.0.10 | 03:54 |
| sdake | sbezverk said he told me what dns_name was yesterday - although it is not in the logs | 03:55 |
| sdake | sbezverk and when i asked he said to refer to the logs - so YMMV :) | 03:55 |
| zhubingbing | http://paste.openstack.org/show/604210/ | 03:57 |
| zhubingbing | sdake https://github.com/openstack/kolla-kubernetes/blob/master/helm/microservice/keystone-create-endpoints-job/templates/keystone-create-endpoints.yaml#L52 | 03:59 |
| zhubingbing | you can read this code, it help you to understand external_vip | 03:59 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Fixing ironic deployment failure on Ubuntu https://review.openstack.org/449803 | 04:00 |
| spsurya | BRB meeting time | 04:03 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Fixing ironic deployment failure on Ubuntu https://review.openstack.org/449803 | 04:05 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Fixing ironic deployment failure on Ubuntu https://review.openstack.org/449803 | 04:10 |
| *** eaguilar has quit IRC | 04:10 | |
| duonghq | zhubingbing, grafana have trouble w/ gate but not local build? | 04:15 |
| zhubingbing | emm | 04:15 |
| *** gkadam has joined #openstack-kolla | 04:15 | |
| duonghq | I can build it successfully | 04:16 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Fixing ironic deployment failure on Ubuntu https://review.openstack.org/449803 | 04:19 |
| openstackgerrit | zhubingbing proposed openstack/kolla master: [WIP] Test kolla gate https://review.openstack.org/450074 | 04:20 |
| zhubingbing | I mean grafana.repo is a bit of a problem, but it is able to build a mirror image successfully | 04:20 |
| duonghq | sure, | 04:28 |
| *** lamt has quit IRC | 04:28 | |
| *** unicell has quit IRC | 04:30 | |
| *** unicell has joined #openstack-kolla | 04:31 | |
| zhubingbing | but now it is look good | 04:33 |
| duonghq | did you wonder why in official doc, we have no el/7/? | 04:35 |
| duonghq | just el/6/ | 04:35 |
| *** yangyapeng has quit IRC | 04:37 | |
| *** caoyuan has quit IRC | 04:37 | |
| *** caoyuan has joined #openstack-kolla | 04:38 | |
| *** yangyapeng has joined #openstack-kolla | 04:38 | |
| sdake | night all | 04:39 |
| zhubingbing | bye sdake | 04:46 |
| duonghq | bye sdake | 04:47 |
| *** dixiaoli has joined #openstack-kolla | 04:48 | |
| *** dixiaoli has quit IRC | 04:49 | |
| *** dixiaoli has joined #openstack-kolla | 04:50 | |
| *** mdnadeem_ has joined #openstack-kolla | 04:51 | |
| *** dixiaoli has quit IRC | 04:52 | |
| *** jrobinson has quit IRC | 04:55 | |
| *** jrobinson has joined #openstack-kolla | 05:04 | |
| *** iceyao has quit IRC | 05:06 | |
| *** japestinho has quit IRC | 05:10 | |
| spsurya | sdake: night | 05:23 |
| duonghq | zhubingbing, seem that situation become worse | 05:24 |
| zhubingbing | ；） | 05:24 |
| zhubingbing | https://review.openstack.org/#/c/450074/ | 05:24 |
| zhubingbing | duonghq can u look it ？ | 05:24 |
| duonghq | I just read this log | 05:25 |
| zhubingbing | i don；t know what‘s happen | 05:25 |
| duonghq | not good | 05:25 |
| duonghq | it's bad, indeed | 05:25 |
| zhubingbing | hmm | 05:26 |
| zhubingbing | ;) | 05:26 |
| *** dixiaoli has joined #openstack-kolla | 05:29 | |
| *** lamt has joined #openstack-kolla | 05:34 | |
| *** caowei has quit IRC | 05:42 | |
| *** caowei has joined #openstack-kolla | 05:42 | |
| *** targon has joined #openstack-kolla | 05:47 | |
| *** rwsu has quit IRC | 05:47 | |
| *** skramaja has joined #openstack-kolla | 05:48 | |
| *** caowei has quit IRC | 05:49 | |
| *** MarginHu has joined #openstack-kolla | 06:03 | |
| openstackgerrit | jimmygc proposed openstack/kolla-ansible master: Add vSphere support to nova_compute https://review.openstack.org/449529 | 06:09 |
| *** jaosorior has joined #openstack-kolla | 06:13 | |
| *** lamt has quit IRC | 06:20 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible master: Enable timeout feature when connecting ceph in cinder https://review.openstack.org/450087 | 06:25 |
| *** lamt has joined #openstack-kolla | 06:26 | |
| *** magicboiz has joined #openstack-kolla | 06:28 | |
| *** manheim has joined #openstack-kolla | 06:37 | |
| duonghq | which is Dell Ironic CI? | 06:40 |
| jaosorior | duonghq: no idea. I've seen that message pop up in a lot of different repos | 06:41 |
| jaosorior | from Tripleo, to puppet repos, and now in kolla | 06:41 |
| jaosorior | I guess someone tried putting it up as third party CI and that didn't go well | 06:41 |
| duonghq | hmm, thank jaosorior | 06:41 |
| duonghq | jaosorior, about this ps: https://review.openstack.org/#/c/449039/2/docker/cinder/cinder-api/Dockerfile.j2@24 | 06:44 |
| duonghq | give the Dockerfile context | 06:44 |
| duonghq | I don't think the file is already existed | 06:44 |
| jaosorior | duonghq: what do you mean by give context? | 06:44 |
| duonghq | the Dockerfile | 06:45 |
| duonghq | I don't think the ports.conf is existed (if in this case, the file is populated file package maintainer before?) | 06:45 |
| duonghq | tbh | 06:45 |
| jaosorior | duonghq: right, it's part of the apache package | 06:45 |
| duonghq | jaosorior, thanks | 06:45 |
| openstackgerrit | jimmygc proposed openstack/kolla-ansible master: Add VMware NSXV support to neutron https://review.openstack.org/450092 | 06:46 |
| jaosorior | duonghq: it's the same issue with RHEL based systems, that some packages for modules also deploy "example" configurations that ultimately try to listen on ports. Which is why I had to do https://review.openstack.org/#/c/446911/ initially | 06:46 |
| duonghq | jaosorior, got it | 06:47 |
| jaosorior | duonghq: thanks for the review | 06:48 |
| duonghq | np :) | 06:49 |
| jaosorior | duonghq: can you also check out these two https://review.openstack.org/#/c/449040 https://review.openstack.org/#/c/446911/ when you get time? | 06:49 |
| duonghq | jaosorior, I'm waiting for the gate | 06:50 |
| *** satyar has joined #openstack-kolla | 06:50 | |
| duonghq | the error in gate-kolla-dsvm-deploy-ubuntu-binary-ubuntu-xenial-nv is painful | 06:50 |
| duonghq | I saw it a few times | 06:50 |
| jaosorior | duonghq: what's the error about? | 06:50 |
| jaosorior | duonghq: seems quite transcient | 06:51 |
| *** caoyuan has quit IRC | 06:51 | |
| duonghq | http://logs.openstack.org/11/446911/8/check/gate-kolla-dsvm-deploy-ubuntu-binary-ubuntu-xenial-nv/e3720bb/console.html#_2017-03-24_08_17_21_752309 | 06:51 |
| duonghq | Jeffrey4l, told me once | 06:51 |
| duonghq | but I cannot recall atm :( | 06:51 |
| *** lamt has quit IRC | 06:53 | |
| openstackgerrit | jimmygc proposed openstack/kolla-ansible master: Add VMware NSXV support to neutron https://review.openstack.org/450092 | 06:57 |
| *** jaosorior has quit IRC | 06:58 | |
| openstackgerrit | jimmygc proposed openstack/kolla-ansible master: Add VMware NSXV support to neutron https://review.openstack.org/450092 | 07:05 |
| openstackgerrit | huwh proposed openstack/kolla-kubernetes master: fix 'raw_doc' doesn`t assignment error when show configmap`s template https://review.openstack.org/384174 | 07:07 |
| *** jaosorior has joined #openstack-kolla | 07:18 | |
| openstackgerrit | jimmygc proposed openstack/kolla-ansible master: Add vSphere support to nova_compute https://review.openstack.org/449529 | 07:18 |
| *** jrobinson has quit IRC | 07:19 | |
| *** pcaruana has joined #openstack-kolla | 07:20 | |
| MarginHu | hi , I want to connect mysql , but I found failed. | 07:28 |
| MarginHu | It hanged when I run "docker exec $container_id mysql -u ironic -h -p" and enter the password. | 07:29 |
| *** japestinho has joined #openstack-kolla | 07:32 | |
| *** manheim has quit IRC | 07:34 | |
| *** chas has joined #openstack-kolla | 07:41 | |
| *** caowei has joined #openstack-kolla | 07:42 | |
| *** Serlex has joined #openstack-kolla | 07:47 | |
| *** klindgren_ has quit IRC | 07:48 | |
| *** klindgren_ has joined #openstack-kolla | 07:49 | |
| duonghq | Jeffrey4l, ping | 07:53 |
| openstackgerrit | Merged openstack/kolla master: Prevent apache from listening on port 80 https://review.openstack.org/449039 | 07:54 |
| *** manheim has joined #openstack-kolla | 07:56 | |
| openstackgerrit | jimmygc proposed openstack/kolla-ansible master: Split globals.yml into mutiple part by service https://review.openstack.org/450106 | 07:58 |
| *** dixiaoli has quit IRC | 07:59 | |
| openstackgerrit | jimmygc proposed openstack/kolla-ansible master: Add vSphere support to nova_compute https://review.openstack.org/449529 | 08:00 |
| *** matrohon has joined #openstack-kolla | 08:04 | |
| *** dixiaoli has joined #openstack-kolla | 08:05 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: ironic-pxe: handle non-x86 architectures https://review.openstack.org/434817 | 08:05 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: debian: enable all images enabled for Ubuntu https://review.openstack.org/432787 | 08:05 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: Add support for non-x86 architectures (aarch64, ppc64le) https://review.openstack.org/430940 | 08:05 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: openstack-base/kolla-toolbox: use mariadb-devel for CentOS and !x86-64 https://review.openstack.org/444832 | 08:05 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: nova-libvirt: handle ppc64le https://review.openstack.org/434810 | 08:05 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: ceph: move ceph-fuse package to cephfs-fuse image https://review.openstack.org/440634 | 08:05 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: openstack-base: Percona-Server is x86-64 only https://review.openstack.org/449965 | 08:05 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: kubernetes: disable for architectures other than x86-64 https://review.openstack.org/445638 | 08:05 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: nova-compute: handle rtslib(-fb) for debian-binary builds too https://review.openstack.org/435941 | 08:05 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: debian: move to stretch https://review.openstack.org/434453 | 08:05 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: mariadb: handle lack of external repos on non x86 https://review.openstack.org/450111 | 08:05 |
| *** dixiaoli has quit IRC | 08:07 | |
| *** dixiaoli has joined #openstack-kolla | 08:08 | |
| duonghq | seem that we should return with gate tomorrow | 08:08 |
| *** dixiaoli has quit IRC | 08:08 | |
| openstackgerrit | Dennis Mueller proposed openstack/kolla-ansible master: Added neutron metering-agent support https://review.openstack.org/445858 | 08:09 |
| duonghq | jaosorior, hope that somebody can fix the gate, or it can self healing | 08:09 |
| openstackgerrit | jimmygc proposed openstack/kolla-ansible master: Add VMware NSXV support to neutron https://review.openstack.org/450092 | 08:13 |
| *** gfidente has joined #openstack-kolla | 08:16 | |
| *** gfidente has quit IRC | 08:16 | |
| *** gfidente has joined #openstack-kolla | 08:16 | |
| *** athomas has joined #openstack-kolla | 08:16 | |
| *** dixiaoli has joined #openstack-kolla | 08:16 | |
| *** jmccarthy has joined #openstack-kolla | 08:20 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: ironic-pxe: handle non-x86 architectures https://review.openstack.org/434817 | 08:22 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: openstack-base/kolla-toolbox: use mariadb-devel for CentOS and !x86-64 https://review.openstack.org/444832 | 08:22 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: ceph: move ceph-fuse package to cephfs-fuse image https://review.openstack.org/440634 | 08:22 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: openstack-base: Percona-Server is x86-64 only https://review.openstack.org/449965 | 08:22 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: kubernetes: disable for architectures other than x86-64 https://review.openstack.org/445638 | 08:22 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: nova-compute: handle rtslib(-fb) for debian-binary builds too https://review.openstack.org/435941 | 08:22 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: mariadb: handle lack of external repos on non x86 https://review.openstack.org/450111 | 08:22 |
| *** tovin07_ has joined #openstack-kolla | 08:22 | |
| hrw | morning | 08:25 |
| *** bjolo has joined #openstack-kolla | 08:26 | |
| *** manheim has quit IRC | 08:26 | |
| *** manheim has joined #openstack-kolla | 08:27 | |
| nea1 | hi, where should the "/etc/neutron/plugins/ml2/openvswitch_agent.ini" be? I can't find it on any of my nodes (`find / -name "*openvswitch*"`), my guess was it should be in /etc/kolla/neutron-openvswitch-agent | 08:32 |
| *** manheim has quit IRC | 08:32 | |
| *** openstackgerrit has quit IRC | 08:33 | |
| *** manheim has joined #openstack-kolla | 08:39 | |
| *** caoyuan has joined #openstack-kolla | 08:40 | |
| *** jascott1_ has quit IRC | 08:42 | |
| *** caowei has quit IRC | 08:43 | |
| *** chas has quit IRC | 08:44 | |
| *** mgoddard has joined #openstack-kolla | 08:50 | |
| *** openstackgerrit has joined #openstack-kolla | 08:51 | |
| openstackgerrit | jimmygc proposed openstack/kolla-ansible master: Add vSphere support to nova_compute https://review.openstack.org/449529 | 08:51 |
| openstackgerrit | jimmygc proposed openstack/kolla-ansible master: Add VMware NSXV support to neutron https://review.openstack.org/450092 | 08:52 |
| openstackgerrit | huwh proposed openstack/kolla master: check mariadb galera status in every loop. https://review.openstack.org/450131 | 09:04 |
| Serlex | ping jimmygc | 09:08 |
| *** pbandark has joined #openstack-kolla | 09:10 | |
| *** dixiaoli has quit IRC | 09:18 | |
| *** dixiaoli has joined #openstack-kolla | 09:19 | |
| duonghq | morning hrw | 09:22 |
| openstackgerrit | jimmygc proposed openstack/kolla-ansible master: Add VMware NSXV support to neutron https://review.openstack.org/450092 | 09:27 |
| *** egonzalez has joined #openstack-kolla | 09:31 | |
| *** caoyuan has quit IRC | 09:32 | |
| *** caoyuan has joined #openstack-kolla | 09:36 | |
| *** caoyuan has quit IRC | 09:40 | |
| *** chas has joined #openstack-kolla | 09:44 | |
| *** MarginHu has quit IRC | 09:45 | |
| *** MarginHu has joined #openstack-kolla | 09:45 | |
| *** MarginHu has quit IRC | 09:47 | |
| *** caoyuan has joined #openstack-kolla | 09:48 | |
| *** chas has quit IRC | 09:49 | |
| *** yingjun has quit IRC | 09:49 | |
| *** sambetts_ is now known as sambetts | 09:51 | |
| *** pbandark is now known as pbandark`brb | 09:55 | |
| *** iceyao has joined #openstack-kolla | 09:56 | |
| *** pbandark`brb has quit IRC | 09:59 | |
| *** bmace has quit IRC | 10:02 | |
| *** iceyao has quit IRC | 10:02 | |
| *** bmace has joined #openstack-kolla | 10:02 | |
| *** eaguilar has joined #openstack-kolla | 10:05 | |
| *** dixiaoli has quit IRC | 10:06 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla master: Alphabetize packages https://review.openstack.org/450150 | 10:08 |
| *** manheim has quit IRC | 10:10 | |
| *** dixiaoli has joined #openstack-kolla | 10:11 | |
| *** daidv has quit IRC | 10:15 | |
| *** eaguilar has quit IRC | 10:17 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla master: Alphabetize packages https://review.openstack.org/450150 | 10:22 |
| *** pbourke has quit IRC | 10:23 | |
| *** pbourke has joined #openstack-kolla | 10:24 | |
| *** tovin07 has quit IRC | 10:27 | |
| *** tovin07_ has quit IRC | 10:27 | |
| *** satyar has quit IRC | 10:36 | |
| *** deadnull has quit IRC | 10:37 | |
| *** dixiaoli has quit IRC | 10:37 | |
| *** cuongnv has quit IRC | 10:38 | |
| *** dixiaoli has joined #openstack-kolla | 10:38 | |
| *** manheim has joined #openstack-kolla | 10:45 | |
| *** manheim has quit IRC | 10:46 | |
| *** manheim has joined #openstack-kolla | 11:01 | |
| *** piotrkopec has joined #openstack-kolla | 11:02 | |
| piotrkopec | Hi people! Could you please give me any advice which OpenStack deployment method should I use to deploy OpenStack on Kubernetes: Kolla, OpenStack-helm or stackanetes? | 11:04 |
| *** zhubingbing has quit IRC | 11:05 | |
| *** dixiaoli has quit IRC | 11:06 | |
| *** piotrkopec has quit IRC | 11:10 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla master: Remove gnocchi-carbonara package https://review.openstack.org/450171 | 11:15 |
| *** dave-mccowan has joined #openstack-kolla | 11:18 | |
| hrw | what is this 'Dell Ironic CI' at gerrit? all messages it generates are "unable to merge" like ones | 11:19 |
| hrw | egonzalez: yum change gave me several failures - going by them one by one | 11:20 |
| egonzalez | hrw no idea, just saw that error | 11:20 |
| jaosorior | hrw, egonzalez http://lists.openstack.org/pipermail/openstack-infra/2017-March/005261.html | 11:21 |
| hrw | egonzalez: sure. and thanks for fixing | 11:21 |
| hrw | jaosorior: thx | 11:21 |
| thomas_oneill | hey how do you guys get your commits linked to your github usernames on the github commit history? | 11:21 |
| egonzalez | cool, jaosorior thx | 11:21 |
| thomas_oneill | somehow your launchpad is linked to your github? | 11:21 |
| * hrw do not care much about github | 11:22 | |
| egonzalez | thomas_oneill no idea, i use the same email for both and also launchpad id is the same | 11:22 |
| jaosorior | thomas_oneill: in github.com click your image. and click Settings > Emails | 11:22 |
| thomas_oneill | wow it worked right away, tyvm jaosorior ! :) | 11:24 |
| jaosorior | thomas_oneill: no biggie | 11:24 |
| *** dixiaoli_ has joined #openstack-kolla | 11:25 | |
| jaosorior | Does anybody know what's going on in kolla CI? Seems that he oraclelinux jobs are quite flaky. | 11:25 |
| *** zhubingbing has joined #openstack-kolla | 11:29 | |
| *** iceyao has joined #openstack-kolla | 11:30 | |
| *** zhurong has joined #openstack-kolla | 11:30 | |
| openstackgerrit | Zeyu Zhu proposed openstack/kolla-ansible master: Remove the redundance variable https://review.openstack.org/448433 | 11:34 |
| *** jascott1_ has joined #openstack-kolla | 11:35 | |
| *** jascott1_ has quit IRC | 11:39 | |
| *** egonzalez has quit IRC | 11:40 | |
| pbourke | jaosorior: can you send me an example? | 11:41 |
| jaosorior | pbourke: https://review.openstack.org/#/c/446911/ | 11:42 |
| *** dixiaoli_ has quit IRC | 11:44 | |
| *** rwallner has joined #openstack-kolla | 11:45 | |
| *** rwallner_ has joined #openstack-kolla | 11:46 | |
| pbourke | jaosorior: seems just unlucky network wise, will try a recheck | 11:47 |
| jaosorior | :( | 11:47 |
| jaosorior | pbourke: thanks | 11:47 |
| *** dixiaoli has joined #openstack-kolla | 11:48 | |
| *** rwallner has quit IRC | 11:49 | |
| pbourke | jaosorior: seems to now have a merge conflict unfortunately | 11:51 |
| jaosorior | pbourke: bummer... I' | 11:52 |
| jaosorior | I' | 11:52 |
| jaosorior | I'll fix it | 11:52 |
| *** manheim has quit IRC | 11:52 | |
| jaosorior | pbourke: oh wait, that wasn't an actual merge conflict | 11:53 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/kolla master: RHEL: Add mod_ssl for services running over httpd https://review.openstack.org/446911 | 11:53 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/kolla master: Prevent apache from listening on port 80 for heat-base https://review.openstack.org/449040 | 11:53 |
| jaosorior | pbourke: see http://lists.openstack.org/pipermail/openstack-infra/2017-March/005261.html | 11:53 |
| pbourke | jaosorior: ah, thanks, was wondering what 'dell ironic ci' was | 11:53 |
| pbourke | jaosorior: will see what recheck comes back with | 11:53 |
| *** bjolo has quit IRC | 12:05 | |
| *** erlon has joined #openstack-kolla | 12:11 | |
| *** dave-mccowan has quit IRC | 12:12 | |
| *** zhurong has quit IRC | 12:12 | |
| *** manheim has joined #openstack-kolla | 12:17 | |
| *** dixiaoli has quit IRC | 12:18 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Fixing ironic deployment failure on Ubuntu https://review.openstack.org/449803 | 12:21 |
| *** satyar has joined #openstack-kolla | 12:22 | |
| *** schwicht has joined #openstack-kolla | 12:24 | |
| *** gkadam has quit IRC | 12:25 | |
| *** yangyapeng has quit IRC | 12:27 | |
| manheim | hello, inc0 around? | 12:31 |
| kolla-slack | <portdirect> Morning kolla o/ | 12:33 |
| kolla-slack | <portdirect> Sdake, you should be able to run 'nslookup kubernetes' from pretty much anywhere in your cluster. If you want an explanation as to what this command does just hit me up. | 12:35 |
| sdake | sup portdirect | 12:35 |
| sdake | ya i know what nslookup does, not sure who populates kubernetes in the dns server | 12:35 |
| kolla-slack | <portdirect> The kubernetes DNS pod | 12:36 |
| sdake | that would be kube-dns? | 12:36 |
| kolla-slack | <portdirect> It monitors endpoints and creates A records accordingly | 12:36 |
| sdake | i see | 12:36 |
| kolla-slack | <portdirect> Yup kube-dns is the fella | 12:36 |
| sdake | tha tis configred by a systemd file | 12:37 |
| sdake | and the systemd file needs to be hand modified to work properly | 12:37 |
| sdake | which probably explains why nslookup kubernetes is not working | 12:37 |
| kolla-slack | <portdirect> Well that totally depends on how you deployed k8s | 12:37 |
| kolla-slack | <portdirect> If using kubeadm its very unlikely you'll need to modify any systemd units | 12:38 |
| kolla-slack | <portdirect> Have a look in /etc/kubernetes/manifests | 12:38 |
| sdake | portdirect i deployed as follows: (reverse-i-search)`kubeadm': sudo kubeadm init --pod-network-cidr=10.1.0.0/16 --service-cidr=10.3.0.0/16 | 12:38 |
| sdake | which means i need to set dns server in kubelet.conf to 10.3.0.10 | 12:39 |
| sdake | pretty sure I made this determination before going to bed last night although didn't have a chance to test it yet | 12:39 |
| kolla-slack | <portdirect> Right, not sure if kubeadm tweaks the drop-in but it should I think | 12:40 |
| sdake | it does not | 12:40 |
| sdake | but it should | 12:40 |
| kolla-slack | <portdirect> If not then we should fix that | 12:40 |
| sdake | kfox1111 said he submitted an issue on the topic | 12:41 |
| kolla-slack | Action: portdirect makes note to investigate when on lunch | 12:41 |
| *** davidsha has joined #openstack-kolla | 12:41 | |
| *** bdek has joined #openstack-kolla | 12:41 | |
| manheim | guys, i wrote an article on kolla, maybe you can take a look and let me know what you think about | 12:42 |
| manheim | https://cloudbase.it/openstack-kolla-hyper-v/ | 12:42 |
| *** MarisK has joined #openstack-kolla | 12:44 | |
| sdake | manheim good article dude :) | 12:44 |
| MarisK | Hi, guys! Does anyone have experience with external ceph in Kolla? I am getting a libvirt error when starting up an instance: https://pastebin.com/WtYimdPa | 12:45 |
| MarisK | Both nova and cinder secrets are defined in Libvirt | 12:45 |
| manheim | skake thank you, looking forward to do more, intergrate the hyperv part into kolla code, maybe we can figure something else, i know inc0 was interested in this | 12:46 |
| pbourke | manheim: cool! | 12:46 |
| sdake | manheim ought to tweet about it so people can find it :) | 12:46 |
| manheim | that would be great, thanks | 12:47 |
| sdake | manheim you ought to tweetabout it :) | 12:47 |
| manheim | lol, forgot to put kolla # in my tweet, damn tweeter won't let me edit it, i'll just retweet :D | 12:48 |
| manheim | done | 12:50 |
| *** mliima has joined #openstack-kolla | 12:50 | |
| sdake | got a tweet link? | 12:50 |
| manheim | https://twitter.com/DomnulDan/status/846343584882311168 | 12:51 |
| *** rwallner_ has quit IRC | 12:51 | |
| *** srwilkers has joined #openstack-kolla | 12:51 | |
| srwilkers | morning | 12:51 |
| sdake | morning srwilkers | 12:51 |
| *** rwallner has joined #openstack-kolla | 12:51 | |
| sdake | sup rwallner | 12:52 |
| mliima | morning | 12:52 |
| sdake | pretty sure i've got a good deploy finally | 12:52 |
| srwilkers | have a good weekend sdake ? | 12:52 |
| sdake | sup mliima | 12:52 |
| srwilkers | awesome dude | 12:52 |
| sdake | srwilkers still alive - so good enough :) | 12:52 |
| srwilkers | sdake, always a good way to look at things | 12:52 |
| *** rwallner has quit IRC | 12:52 | |
| *** rwallner_ has joined #openstack-kolla | 12:52 | |
| rwallner_ | hey sdake | 12:52 |
| sdake | rwallner_ the guide is shaping up - it needs a little mor ework; I think I've figured out all the details however | 12:53 |
| manheim | link working ok? | 12:54 |
| sdake | manheim i dunno i haven't looked moment | 12:54 |
| rwallner_ | sorry forgive me, what were talking about :) | 12:55 |
| sdake | rwallner_ hmm - nm :) | 12:55 |
| rwallner_ | happy to help with the guide though haha | 12:56 |
| rwallner_ | maybe someone with similar handle? | 12:56 |
| sdake | more like its 6am | 12:56 |
| rwallner_ | haha no worries | 12:56 |
| openstackgerrit | Merged openstack/kolla-ansible master: Remove signing_dir option https://review.openstack.org/415607 | 12:57 |
| MarisK | Anyone? :( | 13:00 |
| openstackgerrit | Mohammed Naser proposed openstack/kolla stable/newton: Remove signing_dir option https://review.openstack.org/450203 | 13:00 |
| mnaser | dont know how you guys catch those duplicate bugs :D | 13:01 |
| *** mbruzek has joined #openstack-kolla | 13:05 | |
| *** caoyuan has quit IRC | 13:05 | |
| *** goldyfruit has joined #openstack-kolla | 13:05 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Enable timeout feature when connecting ceph in cinder https://review.openstack.org/450087 | 13:08 |
| *** caoyuan has joined #openstack-kolla | 13:10 | |
| *** davidsha has left #openstack-kolla | 13:10 | |
| jaosorior | pbourke: you had a +A here https://review.openstack.org/#/c/449040/3 but I rebased. Mind checking it out again? | 13:12 |
| *** yangyapeng has joined #openstack-kolla | 13:13 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Update infra components list in README.rst https://review.openstack.org/446873 | 13:16 |
| *** manheim has quit IRC | 13:18 | |
| mgoddard | has anyone had any problems updating nova quotas with ocata kolla-ansible? I get the following error: http://paste.openstack.org/show/604299/ | 13:22 |
| mgoddard | seems to be related to nova assuming the use of versionless keystone endpoints based on this tripleo bug: https://bugs.launchpad.net/tripleo/+bug/1667679 | 13:23 |
| openstack | Launchpad bug 1667679 in OpenStack Compute (nova) "Setting quota fails saying admin project is not a valid project" [Low,Confirmed] | 13:23 |
| *** eanylin has quit IRC | 13:24 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Remove nova-network deprecated options https://review.openstack.org/448685 | 13:25 |
| *** minhas has joined #openstack-kolla | 13:29 | |
| *** manheim has joined #openstack-kolla | 13:31 | |
| *** fooliouno has quit IRC | 13:33 | |
| *** m1dev has joined #openstack-kolla | 13:33 | |
| mgoddard | updating the keystone endpoint catalogue to use versionless keystone endpoints fixed the above quota issue. Further testing would be required to determine whether there are any side effects | 13:35 |
| *** jascott1_ has joined #openstack-kolla | 13:36 | |
| *** jtriley has joined #openstack-kolla | 13:36 | |
| *** eaguilar has joined #openstack-kolla | 13:36 | |
| *** bdek has quit IRC | 13:36 | |
| *** eanylin has joined #openstack-kolla | 13:38 | |
| *** jascott1_ has quit IRC | 13:40 | |
| minhas | Hello. using liberty release and facing neutron-server error related to api-paste.ini. someone can help with this ? | 13:41 |
| *** scheuk has joined #openstack-kolla | 13:45 | |
| nea1 | kolla doesn't support SR-IOV does it? | 13:51 |
| *** rwallner_ has quit IRC | 13:55 | |
| *** esharao has joined #openstack-kolla | 13:57 | |
| *** gkadam has joined #openstack-kolla | 14:00 | |
| *** rwallner_ has joined #openstack-kolla | 14:00 | |
| esharao | sdake.. i tried the installation based on review doc on an ubuntu 16.04 VM.. got till the helm init with only a couple of changes.. | 14:01 |
| *** scheuk has quit IRC | 14:01 | |
| openstackgerrit | Merged openstack/kolla stable/newton: ironic-conductor: add missing comma for centos/source build https://review.openstack.org/449535 | 14:01 |
| *** rwallner_ has quit IRC | 14:02 | |
| *** rwallner has joined #openstack-kolla | 14:05 | |
| *** eaguilar has quit IRC | 14:09 | |
| *** MarisK has quit IRC | 14:10 | |
| *** zhurong has joined #openstack-kolla | 14:11 | |
| *** skramaja has quit IRC | 14:14 | |
| *** iceyao has quit IRC | 14:14 | |
| *** dave-mccowan has joined #openstack-kolla | 14:16 | |
| *** jtriley has quit IRC | 14:16 | |
| *** rwallner has quit IRC | 14:16 | |
| *** manheim has quit IRC | 14:21 | |
| *** manheim has joined #openstack-kolla | 14:24 | |
| *** zhubingbing has quit IRC | 14:28 | |
| *** iceyao has joined #openstack-kolla | 14:29 | |
| *** unicell1 has joined #openstack-kolla | 14:30 | |
| *** unicell has quit IRC | 14:30 | |
| *** athomas has quit IRC | 14:32 | |
| *** gkadam has quit IRC | 14:32 | |
| *** iceyao has quit IRC | 14:34 | |
| *** manheim has quit IRC | 14:38 | |
| *** athomas has joined #openstack-kolla | 14:39 | |
| *** minhas has quit IRC | 14:40 | |
| *** zhurong has quit IRC | 14:41 | |
| *** shashank_t_ has joined #openstack-kolla | 14:42 | |
| *** eaguilar has joined #openstack-kolla | 14:43 | |
| rwellum | esharao: that's great - I am re-trying this morning. I like the cloud.yaml simplification sdake has proposed. | 14:43 |
| openstackgerrit | Mohammed Naser proposed openstack/kolla stable/newton: Remove signing_dir option https://review.openstack.org/450203 | 14:44 |
| esharao | ok.. also i tried with weave net as it did not work with canal for me | 14:44 |
| sdake | rwellum yup - its super simple - although to get things working you have to expand *all* comments to get at the details | 14:45 |
| sdake | rwellum i haven't actually deployed a working kolla-kubernetes in my personal lab (although in my professional work environment lab) | 14:45 |
| sdake | the main thing blocking me at present is the bringup of a proper kubernetes environment - the rest I have done hundreds of times in my professional lab | 14:46 |
| sdake | my professional lab - someone else setup kubernetes | 14:46 |
| sdake | so I am now learning that part | 14:46 |
| sdake | you were right - the CNI part is the hard part | 14:46 |
| rwellum | esharao: that's odd because I could only ever get canal working previously. | 14:46 |
| sdake | although I believe I understand how all that works - unfortunately it is the school hour and I don't have time to document that at this moment | 14:46 |
| rwellum | sdake: yeah - the networking :) | 14:46 |
| sdake | rwellum as my wife says - my co didn't build its empire because networking was easy ;) | 14:47 |
| sdake | rwellum if you have a working kubernetes evnrionment - following hte instructions plsu the addons people have added to the docs should get ou a working deployment | 14:47 |
| esharao | rwellum maybe its my config.. | 14:47 |
| rwellum | sdake: :) | 14:47 |
| sdake | canal requires a special snowflake setup - as canal is prepping for kubernetes 1.6 | 14:48 |
| esharao | i saw kube dns kept on crashing similar to what sdake saw | 14:48 |
| sdake | and they borked their kubernetes 1.5 yaml file | 14:48 |
| *** ccesario has joined #openstack-kolla | 14:48 | |
| sdake | esharao i've got a solution for that - it involves a proper cleanup of your environment | 14:48 |
| rwellum | sdake: ack | 14:48 |
| esharao | Oh.. | 14:48 |
| esharao | it was on a new VM | 14:48 |
| sdake | esharao thats cool then - if you ahve a fresh vm you need a special snowflake canal.yaml which is here: | 14:49 |
| openstackgerrit | Dan Prince proposed openstack/kolla master: Add ironic sudoers file to binary packages as well https://review.openstack.org/450274 | 14:49 |
| *** furlongm has quit IRC | 14:50 | |
| sdake | esharao https://github.com/openstack/kolla-kubernetes/blob/master/tests/bin/setup_canal.sh#L5-L7 | 14:51 |
| esharao | got it.. will try it out on a new VM shortly | 14:52 |
| *** targon has quit IRC | 14:52 | |
| rwellum | Can you just run that script in it's entirety? At the point where you add a CNI? | 14:53 |
| esharao | one more qq - i used the cloud.yaml from sdake but i see that kolla-toolbox is at version 3.0.2 | 14:53 |
| esharao | is that correct? | 14:53 |
| sdake | esharao can you paste your cloud.yaml plz | 14:54 |
| sdake | kolla-toolbox 3.0.2 should work - although I thought all the images in my cloud.yaml in the review were changed to 4.0.0 | 14:54 |
| sdake | I could be mistaken :) | 14:54 |
| sdake | (about the change, not the 3.0.2 should work part) | 14:54 |
| sdake | esharao note - without this special snowflake canal.yaml - the gate was failing as of about 1-2 days ago (when the file was updated) | 14:55 |
| esharao | all: | 14:55 |
| esharao | image_tag: "4.0.0" | 14:55 |
| esharao | kube_logger: false | 14:55 |
| esharao | external_vip: "10.0.3.10" | 14:55 |
| esharao | base_distro: "centos" | 14:55 |
| esharao | install_type: "source" | 14:55 |
| esharao | tunnel_interface: "docker0" | 14:56 |
| esharao | nova: | 14:56 |
| sdake | paste service ftw ;) | 14:56 |
| esharao | sorry.. | 14:56 |
| sdake | ok - time to wrestle my child out of school | 14:56 |
| sdake | i'll bbiaf | 14:56 |
| sdake | out of bed for school that is :) | 14:56 |
| *** jtriley has joined #openstack-kolla | 14:59 | |
| hrw | is someone familiar with kafka image? | 15:00 |
| hrw | I built 206/209 images for centos/source - only bifrost* and kafka are missing. first one looks like "we do not follow openssl development and do not even care to check it's version" while second is fetching some 'random' blob from network into image for !rpm distro | 15:01 |
| *** rwallner has joined #openstack-kolla | 15:02 | |
| *** Serlex has quit IRC | 15:03 | |
| *** yangyapeng has quit IRC | 15:03 | |
| *** rwallner has quit IRC | 15:04 | |
| *** scheuk has joined #openstack-kolla | 15:05 | |
| *** fooliouno has joined #openstack-kolla | 15:06 | |
| *** rwallner has joined #openstack-kolla | 15:07 | |
| *** iceyao has joined #openstack-kolla | 15:08 | |
| *** scheuk has quit IRC | 15:10 | |
| *** scheuk has joined #openstack-kolla | 15:10 | |
| *** iceyao has quit IRC | 15:13 | |
| *** mdnadeem_ has quit IRC | 15:18 | |
| *** lucasxu has joined #openstack-kolla | 15:29 | |
| *** hrw has quit IRC | 15:31 | |
| *** manheim has joined #openstack-kolla | 15:35 | |
| *** klindgren__ has joined #openstack-kolla | 15:35 | |
| *** hrw has joined #openstack-kolla | 15:36 | |
| *** chas has joined #openstack-kolla | 15:38 | |
| *** klindgren_ has quit IRC | 15:38 | |
| *** vhosakot has joined #openstack-kolla | 15:39 | |
| *** vhosakot has quit IRC | 15:39 | |
| *** vhosakot has joined #openstack-kolla | 15:40 | |
| *** chas_ has joined #openstack-kolla | 15:40 | |
| kfox1111 | morning | 15:41 |
| *** chas has quit IRC | 15:42 | |
| sdake | morning kfox1111 | 15:43 |
| sdake | esharao can you paste your cloud.yaml with the toolbox version 3.0.2 | 15:43 |
| *** chas_ has quit IRC | 15:44 | |
| *** jrich523 has quit IRC | 15:44 | |
| *** jrich523 has joined #openstack-kolla | 15:45 | |
| esharao | one sec | 15:45 |
| sdake | esharao here is mine: http://sprunge.us/JJHS | 15:46 |
| sbezverk | kfox1111: good morning | 15:46 |
| *** chas has joined #openstack-kolla | 15:46 | |
| esharao | http://paste.openstack.org/show/604334/ | 15:46 |
| esharao | i just copied your specs from review comments sdake | 15:47 |
| sdake | esharao cool - i may need to update that | 15:47 |
| sdake | esharao i'm not sure which one is more valid | 15:48 |
| *** eaguilar has quit IRC | 15:48 | |
| esharao | hmm.. my kolla-kubernetes did not come up.. it fails for some update-config where it uses the kolla-toolbox:3.0.2 | 15:50 |
| sdake | esharao did your kubernetes come up? | 15:51 |
| sdake | esharao you can test this by starting a busybox and nslookup kubernetes | 15:51 |
| sdake | if that works, your kubernetes is GTG | 15:51 |
| esharao | yes.. some pods came up | 15:51 |
| esharao | like mariadb, rabbitmq | 15:51 |
| esharao | keystone,, | 15:51 |
| esharao | but not all | 15:51 |
| sdake | i mean pre-kolla | 15:51 |
| sdake | esharao can you try this: | 15:51 |
| vhosakot | morning all..raining here in the Boston area... | 15:52 |
| *** Serlex has joined #openstack-kolla | 15:52 | |
| sdake | kubectl run -i -t $(uuidgen) --image=busybox --restart=Never | 15:52 |
| sdake | vhosakot atleast not a blizzard like two weeks ago :) | 15:52 |
| kfox1111 | sdake: there is a test script in test_kube_dns.sh | 15:53 |
| vhosakot | sdake: right... that blizzard was crazy :) | 15:53 |
| kfox1111 | sdake: actually tests all nodes in the cluster, not just one. | 15:53 |
| vhosakot | sdake: so, did you dpeloy on single-VM? | 15:53 |
| sdake | vhosakot i was on the front end of it in boston - glad i made it out before it hit | 15:53 |
| sdake | vhosakot i have got almost everything working | 15:53 |
| vhosakot | sdake: ah, glad you missed it, many planes were affected | 15:53 |
| sdake | vhosakot kubernetes wise - i've got cni working | 15:53 |
| sdake | although my dns is borked and i'm pretty sure i know hwy | 15:54 |
| vhosakot | sdake: are OpenStack pods up? saw your cloud.taml in http://sprunge.us/JJHS.. is that golden.. | 15:54 |
| sdake | vhosakot havven't got that far | 15:54 |
| vhosakot | sdake: cool, did you see my messages and link I sent about dns_name | 15:55 |
| *** schwicht has quit IRC | 15:55 | |
| esharao | sdake: failed to pull image - http://paste.openstack.org/show/604340/ | 15:56 |
| esharao | sdake it may be some proxy issue i have.. checking it | 15:59 |
| vhosakot | sdake: did you add info/explanation about dns_name in the review for guide in https://review.openstack.org/#/c/447356/? I see it has "dns_name: "192.168.1.101". | 16:00 |
| sdake | vhosakot yes i did | 16:00 |
| vhosakot | sdake: cool, which line # ? | 16:00 |
| sdake | vhosakot expand ALL the comments | 16:00 |
| sdake | vhosakot acutally i think sbezverk didn't answer the dns_name value question in the comments | 16:00 |
| vhosakot | sdake: I see dns_name is an input to "helm intstall" | 16:01 |
| *** shashank_t_ has quit IRC | 16:01 | |
| inc0 | good morning | 16:03 |
| *** rwallner has quit IRC | 16:03 | |
| vhosakot | morning inc0 | 16:03 |
| *** klindgren__ is now known as klindgren | 16:04 | |
| klindgren | morning all - Have a question about kolla and neutron-openvswitch-agent | 16:04 |
| klindgren | When using Source as the build target it seems like openvswitch never gets installed: | 16:05 |
| klindgren | https://github.com/openstack/kolla/blob/master/docker/neutron/neutron-openvswitch-agent/Dockerfile.j2 | 16:05 |
| klindgren | because its with in the if install_type == 'binary' | 16:05 |
| inc0 | klindgren: that would be bad;) | 16:07 |
| esharao | sdake fixed my proxy issue.. get the command prompt for busy box | 16:07 |
| openstackgerrit | Bertrand Lallau proposed openstack/kolla-ansible master: Fix Telegraf role reconfigure action https://review.openstack.org/450314 | 16:07 |
| inc0 | however really afair neutron ovs agent doesn't really need ovs package | 16:07 |
| inc0 | it knows how to talk to ovs via socker | 16:07 |
| inc0 | socket | 16:07 |
| inc0 | and ovs is being installed in ovs container | 16:08 |
| klindgren | which results in OVS not being able to work from source at all, because the agent cant use ovs-vsctl or even the improved database connectivity piece as iirc that also requires the db agent or whatever from ovs. | 16:08 |
| klindgren | that assumes ovsdb is running listening on a port no? | 16:08 |
| klindgren | because I attmepted to use unix:/path/to/socket and it still fails in the same maner | 16:09 |
| inc0 | interesting...I mean source works | 16:09 |
| sdake | esharao cool - so next try nslookup kubernetes | 16:09 |
| sdake | nslookup kubernetes | 16:09 |
| kfox1111 | sdake: tools/test_kube_dns.sh | 16:09 |
| inc0 | ovsdb has its own container btw | 16:10 |
| klindgren | inc: I dont see how? I am using source and it blows up because ovs-vsctl doesn't exist in the container? | 16:10 |
| sdake | kfox1111 i heard ya | 16:10 |
| *** mattmceuen has joined #openstack-kolla | 16:10 | |
| inc0 | I always use source... | 16:10 |
| kfox1111 | sdake: ok. | 16:10 |
| esharao | sdake it fails in the lookip | 16:12 |
| inc0 | https://github.com/openstack/kolla/tree/master/docker/openvswitch | 16:12 |
| esharao | *lookup | 16:12 |
| inc0 | we have dedicated ovs containers there | 16:12 |
| sdake | esharao yup - you need to get that working first | 16:12 |
| sdake | esharao before even getting on to kolla :) | 16:12 |
| esharao | hmm ok | 16:12 |
| vhosakot | esharao: yes, "nslookup kubernetes" should work first.. it validates dns in the empty base cluster... | 16:13 |
| esharao | hmm ok will check what is happening in kube-dns | 16:14 |
| sdake | esharao is kube-dns 4/4 with the command kubectl get pods -n kube-system | grep kube-dns | 16:14 |
| sdake | esharao fwiw, this operation fails for me too, and I think I know how to fix it | 16:15 |
| sdake | esharao it requires tearing down my env and I have a meeting in 15 mins | 16:15 |
| sdake | esharao so not sure i will be able to communicate the solution (if it works) prior to then | 16:15 |
| esharao | no issues.. i have one meeting too.. i will probably recreate the VM and try from first again using canal instead of weavenet | 16:16 |
| sdake | vhosakot does have kubernetes dns working though - maybe he has some tips on that | 16:16 |
| sdake | esharao oh your using weave? | 16:16 |
| esharao | yes.. not yet made the change to canal based on your link | 16:16 |
| sdake | i see | 16:17 |
| sdake | ok - well give that a go - who knows if weave works ;) | 16:17 |
| esharao | :) | 16:17 |
| vhosakot | esharao: sdake: not just dns, I check three things before installing kolla-k8s make sure that my empty base cluster is good 1) dns works ("nslookup kubernetes"), 2) none if the CNI pods are restarting, 3) install dashboard pod (to check secrets work with an add-on) | 16:17 |
| sdake | given the state of changes in canal on pending release, weave may be in the same boat | 16:17 |
| esharao | vhosakot- i did get my dashboard up | 16:17 |
| vhosakot | esharao: without dns? | 16:17 |
| sdake | esharao it is possible kolla-kubernetes tanked your network in some way that triggered dns to fail | 16:18 |
| klindgren | inc0, so you run with ovs-db configured to listent on a tcp socket vs's a unix socket? | 16:18 |
| sdake | esharao when you setup with calico with the special uri - make sure to test kubernetes out with 1/2/3 that vhosakot mentioned above | 16:18 |
| klindgren | then I assume you use ovsdb_connection with tcp:<ip>:<port>? | 16:19 |
| *** tovin07 has joined #openstack-kolla | 16:19 | |
| esharao | sure.. will do so | 16:19 |
| inc0 | klindgren: https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/neutron/defaults/main.yml#L22 unix socket | 16:19 |
| vhosakot | esharao: can you access the dashboard GUI when the dns pod is restarting? | 16:20 |
| esharao | sdake.. i think that may be the case. since some pods are up and running.. | 16:20 |
| esharao | vhosakot dns pod is not restarting | 16:20 |
| sdake | esharao is dns 4/4? | 16:20 |
| esharao | yes sdake | 16:20 |
| sdake | esharao thats a good sign | 16:21 |
| sdake | that is step 0/3 :) | 16:21 |
| sdake | esharao if dns works pre-kolla-kubernetes and doesn't post kolla-ubernetes there may be some other misconfiguration in kolla itself | 16:21 |
| esharao | i suspect that as well.. so in the afternoon.. will try to start from fresh and see what happens :D | 16:22 |
| vhosakot | brb 10 | 16:22 |
| *** matrohon has quit IRC | 16:23 | |
| *** eaguilar has joined #openstack-kolla | 16:23 | |
| sdake | kfox1111 what does this error indicate: [sdake@kolla tools]$ kubectl apply -f ./test-dns.yml | 16:24 |
| sdake | The connection to the server localhost:8080 was refused - did you specify the right host or port? | 16:24 |
| sbezverk | sdake: it means kube-apiserver is not listening on 127.0.0.1:8080 the host where you run this command | 16:25 |
| sdake | sbezverk thanks | 16:25 |
| kfox1111 | does 'kubectl get pods' work? | 16:25 |
| sdake | kfox1111 it didn't - i tried it after the apply | 16:26 |
| sdake | i am cleaning up now and starting fresh | 16:26 |
| sdake | bbiaf | 16:26 |
| kfox1111 | k | 16:26 |
| vhosakot | sbezverk: kfox1111 can we re-use the gate scripts to setup dev env? write a wrapper to do everything the gate scripts on dev's machine locally? | 16:28 |
| *** manheim has quit IRC | 16:28 | |
| *** MasterOfBugs has joined #openstack-kolla | 16:28 | |
| kfox1111 | vhosakot: I've been suggesting that. | 16:29 |
| *** duonghq_ has joined #openstack-kolla | 16:29 | |
| duonghq_ | anybody got TSP? | 16:29 |
| *** hachi has quit IRC | 16:29 | |
| vhosakot | kfox1111: I did that 100% successfuly with kolla... re-use all gate scripts to setup local devenv.. it worked great for me... wondering if we can do the same with kolla-kubernetes too? | 16:30 |
| vhosakot | kfox1111: sbezverk the only delta between gate and dev env is IP, hostname, cidr mask, etc, which can be supplied as variable to gate scripts.. | 16:31 |
| kfox1111 | vhosakot: +1. I origionally broke up the scirpts so that most of the stuff could be shared. | 16:31 |
| kfox1111 | would be happy to review patches that made it more generic. | 16:32 |
| kfox1111 | vhosakot: I think there is probably some more stuff then that. the openstack infra does some weird stuff with iptables, etc. | 16:32 |
| *** hachi has joined #openstack-kolla | 16:32 | |
| vhosakot | kfox1111: sbezverk also need to replace references to "/etc/nodepool/" as it does not work for dev env like https://github.com/openstack/kolla-kubernetes/blob/master/tools/setup_kubernetes.sh#L34 | 16:32 |
| kfox1111 | vhosakot: probably the dev env kind of thing though is more of a subset of the gate jobs though, so should be easier. | 16:32 |
| kfox1111 | vhosakot: +1. should be able to wrap them up in a IN_GATE variable or something, then fail back to doing localy things. | 16:33 |
| kfox1111 | I think like 99% of the stuff should be the same. | 16:33 |
| kfox1111 | and continuously tested. | 16:34 |
| vhosakot | kfox1111: does the iptables stuff done by infra show up in gate logs? or is it done after the gate VM is started, but before the gate logs are generate? in which case, I need to ask infra the iptables magic they do in gate. | 16:34 |
| kfox1111 | sbezverk: https://review.openstack.org/#/c/426598/29/tools/setup_gate.sh should I move the functions to setup_gate_common.sh? | 16:34 |
| kfox1111 | vhosakot: we do it in our setup_gate.sh script almost at the begining. | 16:35 |
| vhosakot | kfox1111: cool | 16:35 |
| kfox1111 | vhosakot: they do some iptables work before we get our stuff run, | 16:35 |
| kfox1111 | but its too restrictive to do k8s multinode and some other bits. | 16:35 |
| kfox1111 | so we poke some holes in it. | 16:35 |
| vhosakot | ah ok cool | 16:35 |
| kfox1111 | but the script is kind of gate specific. | 16:35 |
| vhosakot | I'll see if I can re-use gate-scripts to setup dev env, and submit patches.. thanks for the info. | 16:36 |
| kfox1111 | that would be awesome. thanks. :) | 16:36 |
| sbezverk | kfox1111: in common if you think it might be reused by other gate jobs, or into a separate file in it is that job specific thing.. | 16:36 |
| sbezverk | vhosakot: there were several attempt to do it. | 16:37 |
| sbezverk | kolla-kubernetes has already setup_dev_env script | 16:37 |
| klindgren | inc0, what are you using in your mls config under [ovs]? | 16:37 |
| klindgren | [ovs] | 16:37 |
| klindgren | ovsdb_connection = unix:/run/openvswitch/db.sock | 16:37 |
| klindgren | ? | 16:37 |
| vhosakot | sbezverk: what happebed? why did they fail? couldn;t figure our golden configs? | 16:37 |
| sbezverk | vhosakot: instead of reinventing wheel again, maybe you should check that first | 16:37 |
| *** adrian_otto has joined #openstack-kolla | 16:37 | |
| klindgren | ml2* | 16:38 |
| vhosakot | sbezverk: if scripts make things work in gate, I'll take the gate scripts, point it to my local dev env, and add a switch IN_GATE as kfox1111 said.. let me see | 16:38 |
| inc0 | ahh I mislead you | 16:38 |
| inc0 | https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/neutron/templates/ml2_conf.ini.j2#L60 | 16:38 |
| inc0 | tcp | 16:38 |
| sbezverk | vhosakot: as I meantioned before, I am fun of manual labor.. so I have not played with that script and do not know its problems | 16:38 |
| *** rwallner has joined #openstack-kolla | 16:39 | |
| kfox1111 | sbezverk: yeah, all job types should share it. | 16:39 |
| kfox1111 | sbezverk: yeah, but setup_dev_env was a fork I never liked. | 16:39 |
| vhosakot | sbezverk: cool, I see gate script boot nova VM and ping outside from inside the nova VM>. this mean, the gate scripts should be great for compute-kit sanity IMO | 16:39 |
| kfox1111 | sbezverk: I'd like to see it share as much as possible. | 16:39 |
| kfox1111 | otherwise, its much more fragile. | 16:40 |
| sean-k-mooney | has anyone enouchtered this before with one of there ceph OSDs http://paste.openstack.org/show/604349/ | 16:40 |
| vhosakot | well, there is much manual steps in kubeam luckily | 16:41 |
| vhosakot | not much, I meant | 16:41 |
| sbezverk | kfox1111: I am not sure why that forked needed, but if the existing gate scripts can work in local env., great.. | 16:41 |
| kfox1111 | sbezverk: it wasn't needed IMO. it was done for expediency. :/ | 16:41 |
| kfox1111 | now its technical debt. :/ | 16:41 |
| *** jaosorior has quit IRC | 16:41 | |
| kfox1111 | and will break as soon as we finish porting the last bits of ceph over. | 16:42 |
| *** pbandark has joined #openstack-kolla | 16:42 | |
| sean-k-mooney | inc0: by the way do you mind if i rebase https://review.openstack.org/#/c/440834/ ontop of https://review.openstack.org/#/c/416793/ | 16:42 |
| inc0 | sean-k-mooney: yeah I'll do it, will Marcus pick up the patch or should we look for someone else for it? | 16:43 |
| sean-k-mooney | the external ovs patch? | 16:44 |
| *** hachi has quit IRC | 16:44 | |
| inc0 | sean-k-mooney: moving ovs to dedicated role | 16:44 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: WIP: Fernet Token Support https://review.openstack.org/446274 | 16:45 |
| sean-k-mooney | ah well markus was working on it i assumed he still was. | 16:45 |
| inc0 | I mean, my patch isn't needed at all if we move it to role | 16:45 |
| sean-k-mooney | inc0: well we would need to make the inclusion of the role codtional in the sites.yml still correct | 16:45 |
| sean-k-mooney | so that is what your patch would become + docs | 16:46 |
| inc0 | yeha | 16:46 |
| inc0 | easy stuff | 16:46 |
| inc0 | but we need role split patch before | 16:46 |
| sean-k-mooney | yep i taught markus was still working on that but if not i can take a look at it | 16:47 |
| sean-k-mooney | ill ask him | 16:47 |
| mgkwill | sean-k-mooney, inc0 yes still working on it | 16:47 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Simple configmap to raw kubectl https://review.openstack.org/449262 | 16:48 |
| sean-k-mooney | mgkwill: was just looking up your irc nic :) | 16:48 |
| inc0 | ahh sorry then Marcus:) btw good to know your irc handle | 16:48 |
| *** rwsu has joined #openstack-kolla | 16:48 | |
| inc0 | if you need any helm/guidance with it, let me knwo | 16:48 |
| kfox1111 | inc0: hows the genconfig stuff going? | 16:48 |
| *** shashank_t_ has joined #openstack-kolla | 16:49 | |
| sean-k-mooney | inc0 im going to be working on a rewrite of the ovsdpdk stuff over the next 2-3 days. im working form home to focus on it | 16:49 |
| inc0 | kfox1111: need to fix glance:( | 16:49 |
| *** shashank_t_ has quit IRC | 16:50 | |
| inc0 | cool, if we assume external ovs | 16:50 |
| inc0 | it should be simple right? | 16:50 |
| mgkwill | inc0, sean-k-mooney thanks, trying to get gates to pass after debase, but was sidetracked due to deadline on something else | 16:50 |
| mgkwill | *rebase | 16:50 |
| sean-k-mooney | yes assuming external ovs it should be easy. ill need to rebase your patch on markuses to work on that hence my inial quest | 16:51 |
| sean-k-mooney | question | 16:51 |
| *** duonghq_ has quit IRC | 16:51 | |
| inc0 | in fact | 16:51 |
| inc0 | we could create totally external playbook to deploy dpdk | 16:51 |
| inc0 | which is useful regardless of openstack | 16:51 |
| sean-k-mooney | yes that was what i was going to do | 16:52 |
| sean-k-mooney | it would still require the kolla continers but otherwise it would be standalone | 16:52 |
| *** unicell1 has quit IRC | 16:52 | |
| sean-k-mooney | workflwo would be kolla-ansible boostrap-servers -> kolla-ansible deploy-ovs-dpdk -> kolla-ansible deploy | 16:53 |
| *** rwallner has quit IRC | 16:54 | |
| sean-k-mooney | the ovs-dpdk playbook in contrib would be standalone though | 16:54 |
| sean-k-mooney | kolla-ansible deploy-ovs-dpdk would just be syntatic sugar | 16:55 |
| *** mgoddard has quit IRC | 16:59 | |
| *** rwallner has joined #openstack-kolla | 17:00 | |
| *** jmccarthy has quit IRC | 17:00 | |
| openstackgerrit | Dan Prince proposed openstack/kolla master: Add ironic sudoers file to binary packages as well https://review.openstack.org/450274 | 17:02 |
| *** jtriley has quit IRC | 17:03 | |
| *** tovin07 has left #openstack-kolla | 17:03 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Pull container tallballs into the gate https://review.openstack.org/426598 | 17:06 |
| *** skramaja has joined #openstack-kolla | 17:07 | |
| *** gfidente is now known as gfidente|afk | 17:08 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: WIP: Fernet Token Support https://review.openstack.org/446274 | 17:11 |
| *** pbandark has quit IRC | 17:12 | |
| *** pcaruana has quit IRC | 17:12 | |
| *** rwallner has quit IRC | 17:13 | |
| *** jtriley has joined #openstack-kolla | 17:17 | |
| *** mbruzek has quit IRC | 17:18 | |
| *** harlowja has quit IRC | 17:18 | |
| *** schwicht has joined #openstack-kolla | 17:19 | |
| *** harlowja has joined #openstack-kolla | 17:24 | |
| *** unicell has joined #openstack-kolla | 17:24 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Pull container tallballs into the gate https://review.openstack.org/426598 | 17:28 |
| openstackgerrit | Merged openstack/kolla-kubernetes master: Move keystone configuration to kolla-k8s https://review.openstack.org/449254 | 17:29 |
| *** rwallner has joined #openstack-kolla | 17:30 | |
| *** lazyPower is now known as lp|Kubecon | 17:30 | |
| *** schwicht has quit IRC | 17:34 | |
| *** rwallner has quit IRC | 17:35 | |
| *** shashank_t_ has joined #openstack-kolla | 17:39 | |
| klindgren | inc0, figured it out. Had to go back and look at the build stuff | 17:39 |
| klindgren | yum failed to install specific backages into the base image | 17:40 |
| *** lucasxu has quit IRC | 17:40 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: WIP: Fernet Token Support https://review.openstack.org/446274 | 17:40 |
| klindgren | and since its yum, if its installs stuff even though not everything asked for it still says - I did everything all good. | 17:40 |
| klindgren | So silent failure on the image build process. | 17:40 |
| mnaser | klindgren there is a kolla change coming up soon which will avoid this | 17:41 |
| mnaser | klindgren https://review.openstack.org/#/c/449086/ | 17:42 |
| klindgren | mnaser, oh? Does it audit the image after build to ensure it looks the way it should? | 17:42 |
| mnaser | i think the fail is introduced from --setopt skip_missing_names_on_install=False which was removed | 17:42 |
| *** mgoddard has joined #openstack-kolla | 17:43 | |
| *** tonanhngo has joined #openstack-kolla | 17:44 | |
| *** shardy has quit IRC | 17:44 | |
| *** satyar has quit IRC | 17:44 | |
| *** dave-mccowan has quit IRC | 17:46 | |
| *** vhosakot_ has joined #openstack-kolla | 17:46 | |
| *** tonanhngo_ has joined #openstack-kolla | 17:47 | |
| *** tonanhngo has quit IRC | 17:48 | |
| hrw | does someone uses kolla with docker 0.13? | 17:49 |
| hrw | "ERROR:kolla.image.build.base:repository name must be canonical" | 17:49 |
| *** vhosakot has quit IRC | 17:49 | |
| *** tonanhngo has joined #openstack-kolla | 17:50 | |
| *** tonanhngo_ has quit IRC | 17:51 | |
| *** shashank_t_ has quit IRC | 17:56 | |
| *** adrian_otto has quit IRC | 17:57 | |
| *** shashank_t_ has joined #openstack-kolla | 17:57 | |
| sdake | hrw try 1.13.1 | 17:59 |
| sdake | hrw or try 1.12.6 | 18:00 |
| hrw | sdake: it is 1.13.1 (sorry, checked wrong) | 18:00 |
| *** schwicht has joined #openstack-kolla | 18:00 | |
| hrw | 1.12.6 was working | 18:00 |
| sdake | no clue why 1.13.1 doesn't work | 18:00 |
| sdake | 1.13.1 doesn't work with kubernetes either | 18:00 |
| inc0 | klindgren: cool, glad to hear it | 18:01 |
| *** shashank_t_ has quit IRC | 18:01 | |
| *** dave-mccowan has joined #openstack-kolla | 18:01 | |
| *** rwallner has joined #openstack-kolla | 18:02 | |
| hrw | downgraded to 1.12.6 - hope that this version will work for me for next months | 18:06 |
| *** rwallner has quit IRC | 18:07 | |
| *** schwicht has quit IRC | 18:08 | |
| *** lucasxu has joined #openstack-kolla | 18:11 | |
| klindgren | can someone also explain to me what: base_yum_url_packages_override *actually* does? | 18:11 |
| inc0 | right | 18:12 |
| inc0 | so base - name of image | 18:12 |
| rwellum | sdake: trying the latest document, I get to deploying the service level charts and basically all my pods are at init for a goodly period of time. | 18:12 |
| rwellum | https://www.irccloud.com/pastebin/MfTTDBWf/ | 18:12 |
| inc0 | yup_url_packages - https://github.com/openstack/kolla/blob/master/docker/base/Dockerfile.j2#L57 this variable | 18:13 |
| *** satyar has joined #openstack-kolla | 18:13 | |
| hrw | klindgren: allows you to override set of packages installed by yum in base image? | 18:13 |
| inc0 | _override - one of 3 suffixes determining what will happen | 18:13 |
| inc0 | append - add to list, remove - remove from list, override - replace whole list with new list | 18:13 |
| klindgren | ah, kk | 18:14 |
| *** pbandark has joined #openstack-kolla | 18:15 | |
| rwellum | sdake I think I am getting the credentials issue you saw Friday (mariadb) | 18:15 |
| rwellum | https://www.irccloud.com/pastebin/UimOFrRc/ | 18:16 |
| hrw | klindgren: better use _append than _override ;D | 18:16 |
| *** MasterOfBugs has quit IRC | 18:18 | |
| *** sayantan_ has joined #openstack-kolla | 18:20 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Helm Test Ceph Init OSD https://review.openstack.org/450386 | 18:20 |
| *** lamt has joined #openstack-kolla | 18:23 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Pull container tallballs into the gate https://review.openstack.org/426598 | 18:25 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: WIP: Fernet Token Support https://review.openstack.org/446274 | 18:26 |
| *** alop has joined #openstack-kolla | 18:27 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-kubernetes master: Move glance config to kolla-k8s https://review.openstack.org/449755 | 18:37 |
| *** chas has quit IRC | 18:39 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: WIP: Helm Native Values. https://review.openstack.org/449826 | 18:40 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Helm Version File https://review.openstack.org/449814 | 18:40 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Helm Version File https://review.openstack.org/449814 | 18:45 |
| *** mgoddard has quit IRC | 18:47 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Pull container tallballs into the gate https://review.openstack.org/426598 | 18:48 |
| *** pbandark has quit IRC | 18:50 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Helm Test Ceph Init OSD https://review.openstack.org/450386 | 18:56 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Pull container tallballs into the gate https://review.openstack.org/426598 | 19:02 |
| *** schwicht has joined #openstack-kolla | 19:02 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Unpin Canal https://review.openstack.org/450423 | 19:09 |
| *** Manheim has joined #openstack-kolla | 19:12 | |
| *** adrian_otto has joined #openstack-kolla | 19:12 | |
| *** g3ek has quit IRC | 19:12 | |
| *** sambetts is now known as sambetts|afk | 19:13 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: WIP: Fernet Token Support https://review.openstack.org/446274 | 19:13 |
| *** haplo37 has quit IRC | 19:14 | |
| *** erlon has quit IRC | 19:15 | |
| *** g3ek has joined #openstack-kolla | 19:15 | |
| *** eaguilar has quit IRC | 19:15 | |
| *** haplo37 has joined #openstack-kolla | 19:15 | |
| sbezverk | kfox1111: I see you want to unpin canal!?! Have they fixed it to work with 1.5.5? | 19:16 |
| kfox1111 | sbezverk: no, but they are asking for more info. was unpinning and going to hand them a big pile o' logs. :) | 19:17 |
| kfox1111 | issue is: https://github.com/projectcalico/canal/issues/48 | 19:17 |
| sbezverk | kfox1111: ahhh got it, good idea | 19:17 |
| kfox1111 | may have finally figured out the fernet key rotation thing.... | 19:19 |
| kfox1111 | kubectl does a PUT to the resource instead of a POST. | 19:20 |
| kfox1111 | kubectl -v=8 is useful. :) | 19:20 |
| kfox1111 | looks like behavior changed from 1.4 -> 1.5. | 19:20 |
| sdake | rwellum just caught your message | 19:23 |
| sdake | rwellum cn you try something for me plz | 19:23 |
| rwellum | sure sdake | 19:23 |
| sdake | rwellum run this commnd: | 19:24 |
| *** skramaja has quit IRC | 19:24 | |
| kfox1111 | hows this look: https://review.openstack.org/#/c/449814/ ? should be good to go. | 19:26 |
| sdake | kubectl run -i -t $(uuidgen) --image=busybox --restart=Never | 19:27 |
| sdake | rwellum follow that up with nslookup kbuernetes | 19:27 |
| *** shashank_t_ has joined #openstack-kolla | 19:27 | |
| sdake | rwellum i'm not sure i saw that same credentials error you are seeing from docker | 19:27 |
| inc0 | sbezverk: mind re-reviewing? https://review.openstack.org/#/c/449755/ | 19:27 |
| sdake | rwellum after that exit the container and run docker version | 19:27 |
| rwellum | sdake: before I run this, I looked at my watch command again a few pods came up. Some in Running now. It's been running over and hour though. | 19:28 |
| sdake | rwellum i hear ya sounds borked :) | 19:28 |
| sdake | rwellum when i had that happen i had docker 1.13.0 in my system iirc | 19:28 |
| sdake | docker 1.13.0 is busted | 19:28 |
| rwellum | I have 1.12.6 | 19:29 |
| rwellum | https://www.irccloud.com/pastebin/Q3nUsUfU/ | 19:29 |
| sdake | godo to hear - that should be good | 19:29 |
| sbezverk | inc0: done | 19:29 |
| rwellum | sdake: ^^^ nslookup is good. | 19:29 |
| sdake | rwellum your furhter along then I am - your kubernetes environment works ;) | 19:29 |
| rwellum | :) | 19:29 |
| inc0 | thank you good sir | 19:29 |
| sdake | rwellum alhtough i just got off the phone for 2 hours of meetings and have a plan to correct kubernetes | 19:30 |
| sdake | rwellum can you try pulling a docker container that is failing manually with docker cli? | 19:30 |
| sdake | rwellum the containers require no authenitcation to pull from dockerhub | 19:30 |
| sdake | the kolla dockerhub account is not a paid account - there is no way to make thigns "private" | 19:31 |
| rwellum | sdake: hence shouldn't be getting authentication type errors. But I think I am past those now. | 19:33 |
| rwellum | https://www.irccloud.com/pastebin/qb14ycPs/ | 19:33 |
| rwellum | Maybe I am over-committing in a VM like this. | 19:34 |
| *** jtriley has quit IRC | 19:35 | |
| sdake | rwellum how much ram does your vm have? | 19:35 |
| *** srwilkers has quit IRC | 19:36 | |
| sdake | rwellum can you run kubectl get pods -n kolla -> paste | 19:36 |
| rwellum | 32G sdake | 19:37 |
| rwellum | https://www.irccloud.com/pastebin/70pSMISn/ | 19:37 |
| inc0 | btw kfox1111 I might need help if we want to port all genconfig to k8s this week | 19:39 |
| inc0 | I forgot that I have 3 full days of golang training tomorrow-thur :S | 19:39 |
| openstackgerrit | Merged openstack/kolla-kubernetes master: Move glance config to kolla-k8s https://review.openstack.org/449755 | 19:39 |
| kfox1111 | k. not sure I'll be able to, but will see what I can do. | 19:39 |
| inc0 | I'll do nova | 19:40 |
| inc0 | and I'll pick it up Friday | 19:40 |
| kfox1111 | k. | 19:40 |
| inc0 | if you guys want to do cinder or others, would appreciate it | 19:40 |
| inc0 | sdake, vhosakot_ others | 19:40 |
| vhosakot_ | inc0: sup | 19:41 |
| Manheim | inc0 hello | 19:41 |
| inc0 | relatively low hanging fruit, but important | 19:41 |
| *** aimeeu__ has joined #openstack-kolla | 19:41 | |
| sdake | inc0 plate full | 19:41 |
| inc0 | howdy Manheim :) | 19:41 |
| inc0 | sure, just throwing it out there | 19:41 |
| *** aimeeu has quit IRC | 19:41 | |
| sdake | [sdake@kolla ~]$ pastebinit -b http://paste.openstack.org tidy | 19:41 |
| sdake | http://paste.openstack.org/show/604366 | 19:41 |
| *** aimeeu__ is now known as aimeeu | 19:41 | |
| sdake | holy snap - pastebinit works :) | 19:41 |
| inc0 | anyway, if someone would like to pick up a role | 19:42 |
| Manheim | inc0 got the blog post on hyper-v up and running, we talked about it if you remember https://cloudbase.it/openstack-kolla-hyper-v/ | 19:42 |
| Manheim | maybe you can take a look and let me know what you think | 19:42 |
| inc0 | https://review.openstack.org/#/c/449755/ compare this change with kolla-ansible role glance | 19:42 |
| vhosakot_ | inc0: I'm pretty full too.. I'll see if I can do cinder, and let you know. | 19:42 |
| Manheim | and if we could go firther | 19:42 |
| inc0 | Manheim: saw it:) retweeted with pleasure | 19:42 |
| Manheim | oh, thx | 19:42 |
| Manheim | so, we should see how we could streamline a process for this | 19:44 |
| inc0 | it's interesting use case, I'd love to hear more how it's behaving on longer run | 19:44 |
| *** SaMnCo has quit IRC | 19:45 | |
| *** SaMnCo has joined #openstack-kolla | 19:45 | |
| inc0 | vhosakot_: don't worry, just trying to recruit volunteers to kolla-k8s;) | 19:46 |
| vhosakot_ | inc0: you'll be more successfull if we have a nice dev env (which I'm trying to setup and contribute for the docs with others ) :) | 19:46 |
| *** rhallisey has quit IRC | 19:46 | |
| vhosakot_ | gate is kolla-k8s' dev env looks like :) | 19:47 |
| kfox1111 | did something change in the gate? http://logs.openstack.org/74/446274/27/check/gate-kolla-kubernetes-deploy-centos-binary-2-ceph-multi-nv/c555ec4/console.html | 19:47 |
| Manheim | cool, let me know what you have in mind | 19:48 |
| kfox1111 | oh. its inc0's code. | 19:48 |
| inc0 | yay | 19:48 |
| kfox1111 | hmm... | 19:49 |
| *** v1k0d3n has quit IRC | 19:49 | |
| kfox1111 | maybe a bad merge? | 19:49 |
| kfox1111 | seems like its not doing the rm -rf keystone | 19:49 |
| kfox1111 | but trying to manage keystone... | 19:50 |
| inc0 | bad rebase? | 19:50 |
| kfox1111 | was an auto rebase. no conflicts.... | 19:50 |
| sdake | kfox1111 here is what broke canal 1.5 image: quay.io/coreos/etcd:v3.1.4 | 19:50 |
| kfox1111 | but seems to have faild some where. | 19:50 |
| kfox1111 | sdake: running upstream with a switched out etcd works? | 19:50 |
| sdake | kfox1111 not sure - but that really looks suspcious to me | 19:50 |
| kfox1111 | sdake: not proof though. only canal talks to that etcd. | 19:51 |
| *** v1k0d3n has joined #openstack-kolla | 19:51 | |
| sdake | kfox1111 agreed | 19:51 |
| sdake | https://github.com/projectcalico/canal/blame/master/k8s-install/kubeadm/canal.yaml#L165 | 19:51 |
| sdake | https://github.com/projectcalico/canal/blame/master/k8s-install/kubeadm/canal.yaml#L191 | 19:52 |
| kfox1111 | I'd besuspicious of the node one as well. | 19:52 |
| sdake | https://github.com/projectcalico/canal/blame/master/k8s-install/kubeadm/canal.yaml#L306 | 19:52 |
| sdake | lots o changes wrapped up in that yaml :) | 19:52 |
| kfox1111 | yup. | 19:53 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: WIP: Fernet Token Support https://review.openstack.org/446274 | 19:54 |
| inc0 | kfox1111: I removed ssh configs from kube | 19:54 |
| kfox1111 | inc0: keystone? | 19:56 |
| inc0 | yeah | 19:56 |
| kfox1111 | k. thanks. :) | 19:56 |
| inc0 | all keystone-ssh and stuff like that | 19:56 |
| inc0 | so that might be aprt of it | 19:56 |
| kfox1111 | should be fine. we're not using keystone-ssh at all. | 19:56 |
| *** lamt has quit IRC | 19:56 | |
| *** TxGirlGeek has joined #openstack-kolla | 19:58 | |
| *** lamt has joined #openstack-kolla | 19:58 | |
| sdake | kfox1111 does this look correct: | 20:01 |
| sdake | Environment="KUBELET_DNS_ARGS=--cluster-dns=10.0.3.10 --cluster-domain=cluster.local" | 20:01 |
| sdake | for this invocation of kubeadm: | 20:01 |
| *** scheuk has quit IRC | 20:02 | |
| sdake | 1033 sudo kubeadm init --pod-network-cidr=10.1.0.0/16 --service-cidr=10.3.0.0/16 | 20:02 |
| kfox1111 | sdake: I think so. | 20:02 |
| sdake | i'm not sure if cluster dns is in the service cidr or pod cidr | 20:02 |
| kfox1111 | sdake: I think you have to tell systemd to reload the config, and then restart kubelet before it works though. | 20:02 |
| sdake | yup doing that | 20:03 |
| kfox1111 | svc. | 20:03 |
| *** manheim_ has joined #openstack-kolla | 20:03 | |
| sdake | root 27637 1 10 16:03 ? 00:00:00 /usr/bin/kubelet --kubeconfig=/etc/kubernetes/kubelet.conf --require-kubeconfig=true --pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin --cluster-dns=10.0.3.10 --cluster-domain=cluster.local | 20:04 |
| sdake | /opt/cni should be in /var/lib/cni... | 20:04 |
| * sdake groans | 20:04 | |
| kfox1111 | I think the spec is very explicit about it being in /opt/cni | 20:04 |
| sdake | well not my problem to solve | 20:05 |
| sdake | dns still not working | 20:08 |
| * sdake groans | 20:08 | |
| sdake | cluster dns is correct | 20:08 |
| sdake | all pods come up | 20:08 |
| sdake | busybox launches | 20:08 |
| sdake | any suggestions as to what to fix next? | 20:10 |
| sdake | i'm mirroring the gate entirely | 20:10 |
| inc0 | sdake: you need canal? | 20:11 |
| sdake | inc0 apparently that is what the gate uses - yes | 20:11 |
| inc0 | try weave | 20:11 |
| sdake | weave works | 20:11 |
| sdake | although community says weave doesn't work multinode | 20:11 |
| inc0 | I run weave on 3 nodes | 20:12 |
| kfox1111 | so, busybox cant resolve stuff still? | 20:12 |
| inc0 | so I had problems with kubeadm + flannel/canal too | 20:12 |
| sdake | inc0 are you able to nslookup kubernetes from a busybox kubectl launched image? | 20:12 |
| sdake | kfox1111 right | 20:12 |
| inc0 | yeah | 20:12 |
| inc0 | with weave | 20:12 |
| kfox1111 | sdake: and all kube-dns stuff ready? | 20:13 |
| inc0 | with flannel I had some strange iptables issues | 20:13 |
| sdake | 4/4 kfox1111 | 20:13 |
| sdake | no restarts | 20:13 |
| kfox1111 | how about kubectl get svc --all-namespaces |grep dns | 20:13 |
| sdake | kube-system kube-dns 10.3.0.10 <none> 53/UDP,53/TCP 26m | 20:13 |
| kfox1111 | and 'cat /etc/resolv.conf' in the busybox container? | 20:13 |
| kfox1111 | kubectl describe svc kube-dns -n kube-system | 20:14 |
| sdake | nameserver 10.0.3.10 | 20:14 |
| sdake | whisky tango fxoxtrot | 20:14 |
| inc0 | sdake: you know what time is it? | 20:14 |
| inc0 | it's tcpdump time! | 20:15 |
| sdake | i was thinking it was beer-oclock on a monday at 1:15 actually :) | 20:15 |
| kfox1111 | is kube-proxy happy? | 20:15 |
| sdake | http://paste.openstack.org/show/604374 | 20:15 |
| sdake | kfox1111 its running 1/1 28 minutes - not sure what else to check its happiness level | 20:16 |
| kfox1111 | sdake: centos/ubuntu? | 20:17 |
| sdake | centos for life | 20:17 |
| kfox1111 | firewalld/iptables-service checkonfigured off? | 20:18 |
| *** rwallner has joined #openstack-kolla | 20:18 | |
| sdake | roger first step after ra deploy :) | 20:18 |
| kfox1111 | from the host: | 20:18 |
| *** rwallner has quit IRC | 20:18 | |
| kfox1111 | dig @10.0.34.2 google.com | 20:18 |
| sdake | root 27637 1 2 16:03 ? 00:00:20 /usr/bin/kubelet --kubeconfig=/etc/kubernetes/kubelet.conf --require-kubeconfig=true --pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin --cluster-dns=10.0.3.10 --cluster-domain=cluster.local | 20:18 |
| sdake | i see the problem | 20:18 |
| sdake | cluster-dns is improperly set | 20:18 |
| * sdake smacks himself | 20:18 | |
| kfox1111 | ah. that woudl do it. | 20:19 |
| kfox1111 | very easy typo to make. :/ | 20:19 |
| *** rwallner has joined #openstack-kolla | 20:19 | |
| kfox1111 | I asked the kubeadm folks to do that modification tiself. seems silly to make the user do it. | 20:19 |
| *** pbandark has joined #openstack-kolla | 20:19 | |
| sdake | kfox1111 silly is the wrong word | 20:20 |
| kfox1111 | inc0: http://logs.openstack.org/74/446274/28/check/gate-kolla-kubernetes-deploy-centos-binary-2-ceph-nv/21df115/console.html this is failing in your code. any ideas? | 20:20 |
| kfox1111 | sdake: I was trying to be kind. :) | 20:20 |
| sdake | kfox1111 ;) | 20:20 |
| inc0 | kfox1111: looking | 20:21 |
| *** matrohon has joined #openstack-kolla | 20:24 | |
| inc0 | kfox1111: ok managed to reproduce locally, debugging | 20:24 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Helm Test Ceph Init OSD https://review.openstack.org/450386 | 20:25 |
| kfox1111 | inc0: cool. | 20:26 |
| sdake | Name: kubernetes | 20:27 |
| sdake | Address 1: 10.3.0.1 kubernetes.default.svc.cluster.local | 20:27 |
| sdake | snapballs hes on fire! | 20:27 |
| *** manheim_ has quit IRC | 20:27 | |
| kfox1111 | nice. | 20:27 |
| sdake | finally | 20:27 |
| sdake | that only took 20 hours of jerking around | 20:27 |
| inc0 | aaaand got it | 20:27 |
| inc0 | kfox1111: https://github.com/openstack/kolla-kubernetes/blob/master/ansible/roles/keystone/templates/keystone.conf.j2#L33 | 20:28 |
| inc0 | what value should be there? | 20:28 |
| sdake | inc0 before you go on golang traning can you wrap up https://review.openstack.org/#/c/346455/ | 20:28 |
| inc0 | sdake: I asked on #openstack-release where cycle-trailing thing is:/ | 20:29 |
| inc0 | can't find it | 20:29 |
| sdake | ya its gone | 20:29 |
| sdake | i dont know where it went either | 20:29 |
| sdake | if you find out let me knwo :) | 20:29 |
| kfox1111 | inc0: thats a good question. depends on how often you want to rotate, and how much error room you want. | 20:29 |
| inc0 | well, I don't know first thing about fernet | 20:30 |
| kfox1111 | maybe we set it to like 6? | 20:30 |
| inc0 | haha | 20:30 |
| kfox1111 | or 3. | 20:30 |
| inc0 | 42 | 20:30 |
| kfox1111 | thats, to many for sure. :) | 20:30 |
| kfox1111 | its how many keys to allow at once. | 20:30 |
| portdirect | lol - 3 is a pretty good default | 20:30 |
| inc0 | but that's answer to everythign | 20:30 |
| inc0 | so should work | 20:30 |
| kfox1111 | so if its rolling once a day, thats 3 days worth. | 20:31 |
| inc0 | kfox1111: then just hardcode it there | 20:31 |
| inc0 | it works afterwards | 20:32 |
| kfox1111 | k. | 20:33 |
| kolla-slack | <jascott1> inc0 meeting | 20:33 |
| kolla-slack | <jascott1> @inc0 ^ | 20:33 |
| *** TxGirlGeek has quit IRC | 20:34 | |
| sdake | jascott1 meeting! | 20:34 |
| sdake | (of the minds) | 20:34 |
| *** Pavo has joined #openstack-kolla | 20:34 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: WIP: Helm Native Values. https://review.openstack.org/449826 | 20:38 |
| sdake | kfox1111 i think i'm ready to go - and want one last sanity check here | 20:39 |
| sdake | http://paste.openstack.org/show/604377 | 20:39 |
| sdake | kfox1111 my machine's IP address is 192.168.7.101 | 20:39 |
| sdake | does that cloud.yaml look correct? | 20:39 |
| sdake | or do I need to use the pod or service cidr addresses? | 20:40 |
| kfox1111 | sdake: source's is rather untested. but then again, so is most of 4.0. | 20:40 |
| kfox1111 | generally looks ok though. | 20:40 |
| sdake | the ips are correct? | 20:40 |
| kfox1111 | sdake: I think so. | 20:41 |
| sdake | ok here goes :) | 20:41 |
| sdake | http://paste.openstack.org/show/604378 | 20:41 |
| rwellum | Line 9 is bad sdake | 20:43 |
| portdirect | ^^ | 20:43 |
| sdake | indeed thanks rwellum | 20:45 |
| *** shashank_t_ has quit IRC | 20:45 | |
| *** shashank_t_ has joined #openstack-kolla | 20:46 | |
| *** manheim_ has joined #openstack-kolla | 20:47 | |
| kfox1111 | sbezverk: sdake inc0: an easy one: https://review.openstack.org/#/c/449814/ | 20:47 |
| inc0 | here, have a +2 | 20:48 |
| kfox1111 | thx. :) | 20:49 |
| rwellum | sdake: do you need Anthony's placement-api fixes too? Maybe you did it already... | 20:49 |
| *** manheim_ has quit IRC | 20:49 | |
| *** shashank_t_ has quit IRC | 20:50 | |
| *** rhallisey has joined #openstack-kolla | 20:59 | |
| *** gfidente|afk has quit IRC | 21:02 | |
| sdake | rwellum i ran anthony's configmap changes | 21:02 |
| sdake | rwellum I receive the following : nova-compute-9vlqd 0/1 CrashLoopBackOff 7 20m | 21:03 |
| sdake | as well as: cinder-volume-lvm-0pz1q 0/1 Init:0/2 0 | 21:03 |
| sdake | everything else in running state | 21:03 |
| sdake | http://paste.openstack.org/show/604380 | 21:04 |
| sdake | fwiw centos source works in the gate | 21:04 |
| sdake | for 4.0.0 images | 21:04 |
| kfox1111 | weird... looks like kubectl create configmap may not be doing a /_/-/ anymore? | 21:05 |
| inc0 | lmao | 21:05 |
| rwellum | sdake: that's pretty solid results right there... | 21:07 |
| sdake | rwellum more solid then what you have? | 21:07 |
| sdake | rwellum if so, I can share what i did | 21:07 |
| sdake | rwellum although Ineed to take a 10 minte trip to retrieve my daughter from school | 21:07 |
| kfox1111 | ah.... | 21:09 |
| kfox1111 | they fixed it in k8s 1.4 | 21:09 |
| rwellum | Mine's in a world of pain - but I think because I miss-configured. | 21:09 |
| kfox1111 | https://github.com/kubernetes/kubernetes/issues/23722 | 21:09 |
| rwellum | sdake: rather you just updated the doc :) | 21:09 |
| kfox1111 | but kollakube has been continuing to do it. | 21:09 |
| sdake | rwellum you got it | 21:09 |
| sdake | rwellum when i get back, i'll update with the current state - and we can try take 2 :) | 21:09 |
| rwellum | great! | 21:09 |
| *** rwallner has quit IRC | 21:11 | |
| *** Serlex has quit IRC | 21:11 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Simple configmap to raw kubectl https://review.openstack.org/449262 | 21:15 |
| kfox1111 | sbezverk: sdake: inc0: please https://review.openstack.org/#/c/450386/ | 21:16 |
| kfox1111 | another nail in the kollakube coffin. :) | 21:17 |
| *** matrohon has quit IRC | 21:23 | |
| *** dave-mccowan has quit IRC | 21:27 | |
| kfox1111 | score.... fernet token rolling seems to be working now. :) | 21:30 |
| kfox1111 | oh. not quite... | 21:30 |
| sdake | what are these in cloud.yaml: 'resolve_conf_net_host_workaround' and 'enable_resolve_conf_net_host_workaround' | 21:32 |
| kfox1111 | should only need one. seems like one is an older version... | 21:32 |
| kfox1111 | k8s has a lack of feature with reguard to net=host pods and kube-dns. | 21:32 |
| sdake | kfox1111 i understand there is a way to workaround that problem with docker | 21:33 |
| kfox1111 | sdake: yeah, but not with k8s yet | 21:33 |
| sdake | i read it somewhere on the internets | 21:33 |
| kfox1111 | we're working around it with that flag. | 21:33 |
| kfox1111 | and uploading a configmap with the resolv.conf in it. | 21:33 |
| sdake | that flag nees to be true then? | 21:33 |
| kfox1111 | yeah. | 21:33 |
| sdake | ok - one more for cloud.yaml :) | 21:34 |
| sdake | why does it default to false? | 21:34 |
| kfox1111 | the hope was some day it wouldn't be needed.... | 21:34 |
| kfox1111 | that day is taking far far longer then expected. :/ | 21:34 |
| *** lucasxu has quit IRC | 21:34 | |
| kfox1111 | inc0: so, max_active_keys = {{ (groups['keystone'] | length) + 1 }} | 21:35 |
| kfox1111 | am I reading that right? the number of keystone hosts + 1? | 21:35 |
| inc0 | yeah | 21:35 |
| kfox1111 | I'm not sure thats a good way to do that. | 21:35 |
| kfox1111 | its the depth of the allowed keys. | 21:35 |
| *** pbandark has quit IRC | 21:36 | |
| kfox1111 | shouldn't really relate to number of nodes. | 21:36 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: WIP: Fernet Token Support https://review.openstack.org/446274 | 21:36 |
| inc0 | no idea why it's that | 21:36 |
| kfox1111 | fixing it for kolla-kubernetes. but might run the question by whomever submitted it for kolla-ansible. | 21:37 |
| inc0 | hmm it's old patch | 21:39 |
| inc0 | how do you calculate it? | 21:40 |
| kfox1111 | so, how it works is, you have at least one new key, and n-1 older keys that are still valid. | 21:40 |
| *** satyar has quit IRC | 21:40 | |
| kfox1111 | keystone will always use the newest key to issue tokens, | 21:40 |
| kfox1111 | and validate incoming tokens against the n-1 older keys. | 21:40 |
| kfox1111 | so you need enough to allow a rotate to happen, and get copied to all the keystone nodes. | 21:41 |
| inc0 | so I guess n-nodes is worst case scenerio of having n old keys + new one | 21:41 |
| sbezverk | kfox1111: any news on canal? I want to try it on my cluster to see if it breaks anything. | 21:41 |
| *** lucasxu has joined #openstack-kolla | 21:41 | |
| inc0 | by worst case I mean each node has different key | 21:41 |
| kfox1111 | sbezverk: https://github.com/projectcalico/canal/issues/48 | 21:41 |
| kfox1111 | inc0: its more a factor of when each keystone node synced last. | 21:42 |
| kfox1111 | if they are keeping up to date, 2 is more then sufficient. | 21:42 |
| *** MarginHu has joined #openstack-kolla | 21:42 | |
| kfox1111 | if some start lagging for some reason, then more are needed. | 21:42 |
| kfox1111 | so, really depends on if whatever mechanism you came up with to sync the keys runs into syncing issues. | 21:43 |
| kfox1111 | or if you have a node down for a while, then it comes back up without refreshing keys. | 21:43 |
| kfox1111 | in kolla-kuberetes, its going to use secrets. | 21:43 |
| kfox1111 | so takes a couple minutes to go from job -> pods. | 21:43 |
| kfox1111 | and a new copy is fetched everytime a pod is restarted/creted. | 21:43 |
| kfox1111 | so, a very small amount's probably fine. | 21:44 |
| sdake | sbezverk canal WFM | 21:44 |
| sdake | however compute is in a crashloopbackoff | 21:45 |
| sdake | placement api up - also the cell0 configmaps are generated and helm charts launched | 21:45 |
| sdake | sbezverk to get canal to work I had to check out the repo revert the last commit and modify it to match kubeadm parameters | 21:46 |
| sbezverk | sdake: I have canal working for more than 3 month now :) | 21:47 |
| sdake | sbezverk ya - my internal lab is using weave not canal | 21:47 |
| sdake | can someone explain to me why we want to use canal | 21:47 |
| sdake | i have heard straight calico is the way to go from portdirect | 21:47 |
| sdake | less problematic | 21:47 |
| sdake | what advantage does canal bring? | 21:47 |
| *** vhosakot has joined #openstack-kolla | 21:48 | |
| kfox1111 | sdake: my understanding is calico and flannel are going away. | 21:48 |
| kfox1111 | they are merging into canal. | 21:48 |
| sdake | kfox1111 cool - any references or chatter on slack? | 21:48 |
| sbezverk | kfox1111: flannel cannot go away, there must be at least one l2 solution | 21:49 |
| *** vhosakot_ has quit IRC | 21:50 | |
| kfox1111 | https://www.projectcalico.org/canal-tigera/ | 21:50 |
| kfox1111 | sbezverk: I think the plan is to combine binaries into one thing that does it all. but can switch off the parts you dont want. | 21:50 |
| sbezverk | kfox1111: ah ok, as long as there is a choice | 21:50 |
| kfox1111 | sbezverk: are you pinning ironic to v4 in the v3 gate? | 21:53 |
| sbezverk | kfox1111: v4 only | 21:53 |
| sbezverk | it is just the name is wrong | 21:53 |
| kfox1111 | sbezverk: so, if the ps to infra that I'm makeing did an s/3/4/ on the ironic gates, yo uwoudln't be sad? :) | 21:53 |
| sbezverk | kfox1111: I wanted to suggest you to use instead of version numbers | 21:53 |
| sbezverk | -t- and -t-1 in the name | 21:54 |
| kfox1111 | thats going to be a mess whenever something new comes about. | 21:54 |
| sbezverk | then gate job could keep the same name | 21:54 |
| kfox1111 | yeah, but the older versions would shift down, and then the logic woudl be all messed up. | 21:54 |
| kfox1111 | adding a gate job would be easier then trying to deal with the sift? | 21:55 |
| sbezverk | kfox1111: not sure, since switching from one version to another would be just a metter changing image tag | 21:55 |
| kfox1111 | shift? | 21:55 |
| kfox1111 | in theory, yeah. but not yet. :/ | 21:55 |
| kfox1111 | once 1.0.0 hits, much more likelyy. | 21:55 |
| kfox1111 | I'm wondinerg what 5.x containers we will have to create before 1.0.0 lands though. | 21:56 |
| sbezverk | kfox1111: hopefully not | 21:56 |
| sdake | you mean charts? | 21:56 |
| kfox1111 | no, containers. | 21:56 |
| sdake | we have a whole team that creates containers | 21:56 |
| kfox1111 | like, when we add prometheus support. what sidecar containers will we need? etc. | 21:56 |
| sdake | we did say we wouldn't do anything in 1.0.0 that isn't alrady in kolla-asnible | 21:57 |
| kfox1111 | who's we? | 21:57 |
| sdake | at the ptg? | 21:57 |
| kfox1111 | I don't remember that. | 21:57 |
| sdake | moment | 21:58 |
| sdake | https://etherpad.openstack.org/p/kolla-pike-ptg-k8s-release-roadmap | 21:58 |
| sdake | line 44 | 21:58 |
| kfox1111 | sdake: "goal" I agree. | 21:59 |
| sdake | why do you think i'm working on line 31 :) | 21:59 |
| sbezverk | kfox1111: excess in consumption kills memory cells, actually converts them into cell0 ;) | 21:59 |
| kfox1111 | that doesn't mean we will turn down features. | 21:59 |
| kfox1111 | sbezverk: hehe | 21:59 |
| *** vhosakot has quit IRC | 22:00 | |
| *** jrobinson has joined #openstack-kolla | 22:00 | |
| sdake | it isn't for my health - i can tell you that :) | 22:00 |
| sdake | kfox1111 i think instead of expanding scope we should keep it limited to the items we defined and really try to hit the pike deadlines | 22:02 |
| sdake | kfox1111 if we cram every feature into 1.0.0 - we will enver release a 1.0.0 | 22:03 |
| sdake | someone at the ptg said "how does one release a 1.0.0" | 22:03 |
| sdake | I have the answer to that question. | 22:03 |
| sdake | this philosophy among many other techniques is one that gets us to 1.0.0 | 22:04 |
| openstackgerrit | Merged openstack/kolla-kubernetes master: Helm Version File https://review.openstack.org/449814 | 22:04 |
| kfox1111 | sdake: there are parts that align with what I need to get done, and parts that are less important. I understand to kolla's mission, orchestration is important. less so to me though. I only have so much time to contribute. | 22:04 |
| sdake | i've been through the "lets cram everything into 0.91 and into 0..92 and 0.93 etc - for years - 5 years in fact with corosync | 22:04 |
| *** david-lyle has quit IRC | 22:04 | |
| kfox1111 | being able to successfully monitor a deployed openstack is high on the list, for example. | 22:05 |
| sdake | kfox1111 i suppose my point is we aren't going to block on it | 22:05 |
| kfox1111 | right. | 22:05 |
| sdake | if someone wants to crank out the code - more power to em :) | 22:05 |
| kfox1111 | yup. | 22:05 |
| sdake | i would think sorting out upgrades would be higher on your list tho :) | 22:05 |
| sdake | but not inside our head | 22:05 |
| kfox1111 | that one is high up on the list too. | 22:05 |
| *** david-lyle has joined #openstack-kolla | 22:05 | |
| kfox1111 | tahnks for the reminder though... this PS would be a good time to introduce the gate job for that.... | 22:06 |
| sdake | gate job for which | 22:06 |
| kfox1111 | upgrade. | 22:06 |
| sdake | which ps? | 22:07 |
| kfox1111 | the one I'm making to infra | 22:07 |
| sdake | typically its beest practice to create the implementation prior to the gate job :) | 22:07 |
| sdake | otherwise we have gates doing nothign sucking up resources | 22:07 |
| *** esharao has quit IRC | 22:07 | |
| sdake | or use a depends on on the implementation | 22:07 |
| sdake | we have a few of those gates already taht need removal | 22:08 |
| kfox1111 | cant trigger it if its not in the gate. | 22:08 |
| kfox1111 | can be dummied up to do nothing when not implemented. | 22:08 |
| sdake | need some basic implementation | 22:08 |
| kfox1111 | so doesn't use resources. | 22:08 |
| sdake | chicken egg thing - egg always comes before the chikcen - the egg in the case is the implementation no the gate ;) | 22:09 |
| kfox1111 | never done it that way, in all of kolla-kubernetes's past. | 22:09 |
| kfox1111 | infra's never complained. | 22:09 |
| sdake | well its not like i'm going to stop you | 22:10 |
| sdake | do whatyou think is right :) | 22:10 |
| *** fooliouno has quit IRC | 22:10 | |
| kfox1111 | https://review.openstack.org/450471 | 22:12 |
| kfox1111 | usually takes a few days to go through. | 22:13 |
| kfox1111 | there we go... fernet rotation working. | 22:14 |
| kfox1111 | now for some cleanup. | 22:14 |
| kfox1111 | hmm... | 22:15 |
| kfox1111 | ok, so here's an interesting case... | 22:15 |
| kfox1111 | do we want to support !fernet? | 22:16 |
| kfox1111 | if so, we need a conditional in the deps of keystone's service packages. | 22:16 |
| sbezverk | kfox1111: we should build two service packages for keystone | 22:20 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Remove enable_resolve_conf_net_host_workaround https://review.openstack.org/450477 | 22:20 |
| kfox1111 | sbezverk: seems like a bit of overkill for just one extra feature. | 22:20 |
| *** jascott1_ has joined #openstack-kolla | 22:20 | |
| sdake | sbezverk if we do that how does computekit deploy? | 22:21 |
| sbezverk | one normal and one with fernet | 22:21 |
| sdake | jascott1_ yo! | 22:21 |
| *** jascott1_ has quit IRC | 22:21 | |
| kfox1111 | fernet's the thing keystone's really pushing for. do we really care to support !fernet? | 22:21 |
| kfox1111 | its easy to go from !fernet -> fernet. | 22:22 |
| sdake | we want a secure implementation of openstack | 22:22 |
| kfox1111 | sbezverk: https://review.openstack.org/#/c/450386/ please | 22:23 |
| kfox1111 | sdake: +1 | 22:23 |
| sdake | the current key technology from my understanding is not particularly secure | 22:23 |
| sbezverk | kfox1111: people are complaining about complexity.. I think just building two different flavors is less complex than adding another dependency check into the code.. | 22:23 |
| kfox1111 | sbezverk: yeah, but building one is less complex then 2. | 22:23 |
| sdake | sbezverk how to deploy computekit? | 22:23 |
| kfox1111 | sdake: compute kit could conditionally include one or the other. | 22:24 |
| sdake | we face the same problem with the placement api and cells | 22:24 |
| sbezverk | sdake: you have dependency file, you change the name | 22:24 |
| sbezverk | or you really want to make everything one click operation? | 22:24 |
| sdake | we dont want to have people hae to change the charts | 22:24 |
| sdake | its hard to get to one click operation | 22:24 |
| sdake | eve nkolla requires 4 or 5 clicks ;) | 22:24 |
| sbezverk | sdake: dependency yaml is NOT chart | 22:24 |
| sdake | kolla-ansible that is | 22:24 |
| sdake | doesn't that go in the chart? | 22:25 |
| sbezverk | you have no problem changing cloud.yaml, do not see much of a difference for requirements | 22:25 |
| sdake | cloud.yaml is passed via command line | 22:25 |
| sdake | that is sanity to me | 22:25 |
| sdake | is dependency.yaml passed via command line? | 22:25 |
| sdake | santiary that is | 22:26 |
| sdake | sanitary | 22:26 |
| *** gkadam has joined #openstack-kolla | 22:26 | |
| sdake | 3rd try is a charm :) | 22:26 |
| sdake | one service chart per service makes the msot sense | 22:26 |
| sdake | otherwise we will end up wit hendlesss variations of service charts | 22:26 |
| kfox1111 | sdake: dependencies are just helm values. so could be passed through. | 22:27 |
| sbezverk | I am perfectly ok with adding another dependecy, just then do not complain about complexity of logic in charts | 22:28 |
| sdake | who compalined? | 22:29 |
| kfox1111 | just wondering if we need to add the feature condtionally at all? do we care to support uuid/pki tokens when keystone has been strongly recommending everyone get off of them? | 22:29 |
| sdake | kfox1111 i'm with you on that one | 22:29 |
| kfox1111 | (I had to convert all my existing clouds to fernet for performance reasons) | 22:29 |
| sdake | although i care more about nova api placmeent since nova for 4.0.0 needs to work :) | 22:29 |
| kfox1111 | +1. | 22:30 |
| sbezverk | kfox1111: then let's make fenet keystone default | 22:30 |
| sdake | and cell0 whatever the hell that is | 22:30 |
| sbezverk | and keep old way for backward compatibility | 22:30 |
| kfox1111 | keeping old way requerss conditionals though. | 22:30 |
| sdake | we are going to need conditionals for some things i suspect | 22:31 |
| *** jascott1_ has joined #openstack-kolla | 22:31 | |
| sbezverk | kfox1111: that is fine, since sdake is ok with conditionals | 22:31 |
| sdake | e.g. i want spicehtml5proxy instead of novanovncproxy | 22:31 |
| sdake | i dont even know what a conditonal in helm looks like :) | 22:31 |
| kfox1111 | agreed, but not sure it requires conditional jobs with dependencies? | 22:31 |
| kfox1111 | cell0 might... | 22:31 |
| sdake | it would be nice if we had just one config file to rule them all | 22:32 |
| kfox1111 | really the issue is of conditional env's for depenedencies... | 22:32 |
| kfox1111 | maybe the lib values ps will fix it. | 22:32 |
| kfox1111 | sdake: thats kind of the intention of cloud.yaml | 22:32 |
| sdake | kfox1111 yup | 22:33 |
| sdake | kfox1111 what i mean is another "dependencies.yaml" passed in is extra complexity on the user | 22:33 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: WIP: Fernet Token Support https://review.openstack.org/446274 | 22:33 |
| sdake | its typically better to have a simple interface and a complex implementation then a complex interface and a simple implementation | 22:33 |
| sdake | although that needs to be balanced | 22:33 |
| kfox1111 | +1 | 22:33 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Fernet Token Support https://review.openstack.org/446274 | 22:34 |
| sdake | if you end up with a complex interface and a simple interface more layers of abstraction are required to simplify the complexity | 22:34 |
| sdake | rather complex interface+simple implementation | 22:34 |
| sdake | this is really a design constraint of helm imo | 22:34 |
| kfox1111 | yeah | 22:34 |
| sdake | a proper system would have abstractions built in to hide complexity - helm doesn't relaly ahve that | 22:35 |
| sdake | rpm is the same story | 22:35 |
| *** Manheim has quit IRC | 22:36 | |
| kfox1111 | rpms about packaging, not config. | 22:36 |
| sdake | there is config in rpm :) | 22:36 |
| kfox1111 | helm's in a weird place where its packaging, but some config too. | 22:36 |
| *** lamt has quit IRC | 22:36 | |
| kfox1111 | no, there is config files in rpm. but not config of the running instance. | 22:36 |
| sdake | your right helm has more config | 22:37 |
| kfox1111 | kind of a different beast. | 22:37 |
| inc0 | sdake: https://github.com/openstack/releases/blob/master/deliverables/pike/kolla.yaml#L3 | 22:37 |
| kfox1111 | inc0: your keystone patch allowed this to be abandoned: https://review.openstack.org/#/c/447159/ :) | 22:38 |
| kfox1111 | inc0: sdake: second please? https://review.openstack.org/#/c/450386/ | 22:39 |
| inc0 | yay | 22:39 |
| inc0 | done | 22:39 |
| kfox1111 | now on to test-ceph-osd... :) | 22:40 |
| sdake | kfox1111 https://review.openstack.org/#/c/450386/3/helm/microservice/test-ceph-init-osd-job/templates/test-ceph-init-osd-job.yaml | 22:41 |
| sdake | is line 62/66 correct? | 22:41 |
| kfox1111 | sdake: yeah. thats why its test-ceph-... | 22:41 |
| *** m1dev has quit IRC | 22:41 | |
| inc0 | sdake: https://review.openstack.org/#/c/346455/4 | 22:41 |
| sdake | kfox1111 I mean the lack of " | 22:42 |
| kfox1111 | not really designed to be very robust. just enough to be robust for testing. | 22:42 |
| kfox1111 | sdake: not sure I see the lack of a " | 22:42 |
| sdake | kubernetes.io/hostname: {{ $initial_mon }} | 22:42 |
| sdake | - image: "{{ $imageFull }}" | 22:42 |
| sdake | imagePullPolicy: {{ $imagePullPolicy }} | 22:43 |
| kfox1111 | asking if initial_mon and imagePullPolicy shoudl be quoted? | 22:43 |
| sdake | inconsistent - not sure if it matters | 22:43 |
| sdake | right | 22:43 |
| * kfox1111 shrugs. inital_mon is a dns name. so no newlines or anything goofy. | 22:43 | |
| kfox1111 | image pull policy is an enum. | 22:43 |
| kfox1111 | if we really wanted to be clean though,k we could quote it. | 22:44 |
| kfox1111 | though all cases where it was not quoted right, would break it anyway I think. | 22:44 |
| kfox1111 | 6 of one. | 22:44 |
| kfox1111 | something we could put in a blueprint for though. | 22:44 |
| kfox1111 | general code cleanliness thingies. | 22:44 |
| kfox1111 | probably a lot more cases of it. | 22:45 |
| *** manheim has joined #openstack-kolla | 22:45 | |
| openstackgerrit | Merged openstack/kolla-kubernetes master: Helm Test Ceph Init OSD https://review.openstack.org/450386 | 22:46 |
| *** lucasxu has quit IRC | 22:51 | |
| sdake | rwellum you can use git am path-to-patch to apply that placement api fix | 22:54 |
| sdake | which nick is Anthony Lin? | 22:55 |
| sdake | inc0 re https://review.openstack.org/#/c/346455/4 you missed many of my comments | 22:56 |
| sdake | inc0 we don't need anything about liberty in there imo | 22:56 |
| sdake | and also i can't contribute to the backports process | 22:56 |
| sdake | inc0 my day is capped at 24 hours as a result of physics :) | 22:57 |
| *** Pavo has quit IRC | 22:57 | |
| inc0 | sdake: check newer patchset | 22:58 |
| sdake | inc0 wierd not sure where that old one came from :) | 22:59 |
| sdake | inc0 looks good althoug hthe stable policy requires 4-5 people iirc | 22:59 |
| *** Pavo has joined #openstack-kolla | 23:00 | |
| sdake | inc0 also in a previous review bitrot jobs were requested | 23:00 |
| sdake | i hae no idea what that is | 23:00 |
| sdake | although it wasn't blocking the review | 23:00 |
| kfox1111 | sbezverk: is tools/patches/0001-Revert-Fixes-problem-with-kolla-kubernetes-3.0.2-ima.patch still needed? | 23:01 |
| sdake | kfox1111 why yes it is | 23:01 |
| kfox1111 | :/ | 23:01 |
| kfox1111 | inc0: you said you were doing nova next? | 23:02 |
| kfox1111 | inc0: can you roll tools/patches/0001-Revert-Fixes-problem-with-kolla-kubernetes-3.0.2-ima.patch into it? :) | 23:02 |
| sdake | kfox1111 its in master already - i assume inc0 is copying from master | 23:02 |
| inc0 | kfox1111: kk | 23:02 |
| inc0 | yeah | 23:02 |
| inc0 | I'm copying master roles | 23:02 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Fixing ironic deployment failure on Ubuntu https://review.openstack.org/449803 | 23:02 |
| sdake | inc0 might want to double check my assumption - pretty sure that merged in master | 23:02 |
| kfox1111 | k. | 23:02 |
| kfox1111 | not sure. otherwise, wouldn't the pathc fail to apply? | 23:03 |
| sdake | kfox1111 not really sure - i haven't tried that latest peice of code | 23:03 |
| sdake | kfox1111 hence why i suggested double chekcing | 23:03 |
| sbezverk | kfox1111: I think so | 23:03 |
| sdake | yes the patch would merge conflict | 23:04 |
| sbezverk | kfox1111: I do not know how v2.0.0 images will react on placement section | 23:04 |
| *** goldyfruit has quit IRC | 23:04 | |
| kfox1111 | sbezverk: its been ok I think. | 23:05 |
| kfox1111 | I think its been there for a long time. | 23:05 |
| kfox1111 | just an unknown ini section I think to 2.x | 23:05 |
| sbezverk | kfox1111: we can try t remove and see ;) | 23:05 |
| *** jrobinson has quit IRC | 23:06 | |
| sdake | [sdake@kolla kolla-ansible]$ git am ../kolla-kubernetes/tools/patches/0001-Revert-Fixes-problem-with-kolla-kubernetes-3.0.2-ima.patch | 23:06 |
| sdake | Applying: Revert "Fixes problem with kolla-kubernetes 3.0.2 images" | 23:06 |
| sdake | definately not in master | 23:06 |
| * sdake groans | 23:06 | |
| kfox1111 | sdake: fun. | 23:07 |
| kfox1111 | if inc0 can get the nova patch up quickly, we can avoid it all together. | 23:07 |
| sbezverk | sdake: how it could nbe not in master if it got merged right before I pushed placement api ps?? | 23:08 |
| sdake | dunno its not in master | 23:09 |
| sdake | https://review.openstack.org/#/c/444351/ | 23:09 |
| sdake | workflow -1 | 23:09 |
| sbezverk | if you git clone kolla-kuebernetes, you should see this file at that location | 23:10 |
| sdake | sbezverk its not in kolla-ansible | 23:10 |
| sdake | sbezverk hence placement api doesn't work from a git clone outside of the gate | 23:11 |
| sbezverk | it was never meant to be in kolla ansible | 23:11 |
| sdake | sbezverk how should kolla-kubernetes work outside the gate then? | 23:11 |
| sbezverk | Jeffrey4l built it just for kolla-kube | 23:11 |
| kfox1111 | patching kolla-ansible from kolla-kubernetes source doen't seem likea good plan. | 23:11 |
| sbezverk | sdake: same as it works in the gate by applying the patch | 23:11 |
| sdake | no - pretty terrible :) | 23:12 |
| kfox1111 | I'm with sdake. | 23:12 |
| kfox1111 | lets just work on undoing it. | 23:12 |
| sdake | anyway it will be gone soon enough | 23:12 |
| kfox1111 | yeah. | 23:12 |
| sdake | wish i knew where my remote was | 23:12 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Simple configmap to raw kubectl https://review.openstack.org/449262 | 23:12 |
| kfox1111 | sdake: you need a remote for your remote. :) | 23:14 |
| sdake | kfox1111 in fact i bought some :) | 23:15 |
| sdake | called a tile | 23:15 |
| *** caoyuan has quit IRC | 23:15 | |
| sdake | now just need to find the remote to attach it to | 23:16 |
| sdake | can ping it from my phone or laptop :) | 23:16 |
| sdake | phone esy to misplace - laptop harder as i'm on it 20 milliion years a day | 23:16 |
| kfox1111 | sdake: yeah. I snagged a tile as swag somewhere... | 23:16 |
| kfox1111 | very cool. :) | 23:16 |
| *** signed8bit has joined #openstack-kolla | 23:21 | |
| *** manheim has quit IRC | 23:22 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Simple configmap to raw kubectl https://review.openstack.org/449262 | 23:24 |
| *** mattmceuen has quit IRC | 23:30 | |
| *** MarginHu has quit IRC | 23:31 | |
| *** schwicht has quit IRC | 23:31 | |
| sdake | where are the logs storeth in kolla-kubernetes? | 23:32 |
| kfox1111 | in the containers. :/ | 23:33 |
| kfox1111 | fluentd integration is a work in progress. | 23:33 |
| sdake | there must surely be a way to hostmount in kubernetes? | 23:34 |
| sdake | can't we have a short term solution to log output for dev | 23:34 |
| kfox1111 | sure. | 23:34 |
| inc0 | there is | 23:34 |
| kfox1111 | but kubectl exec pod is generally easier then ssh host. | 23:34 |
| sdake | kfox1111 that works great if the container is alive | 23:35 |
| kfox1111 | shoudln't have to touch the host most of the time. | 23:35 |
| sdake | if its in a crashloopbackoff, no bueno | 23:35 |
| inc0 | https://kubernetes.io/docs/concepts/storage/volumes/#hostpath | 23:35 |
| kfox1111 | sdake: thats why we had a debugging feature in kollakube. | 23:35 |
| kfox1111 | helm doesn't have it. :/ | 23:35 |
| inc0 | kfox1111: how did it work really? | 23:35 |
| kfox1111 | it overrode the containers cmd to be '/bin/bash -c "while true; do sleep 500; done" | 23:36 |
| kfox1111 | and then you could go in and kolla_start stuff yourself. | 23:36 |
| inc0 | we can do it in helm too | 23:36 |
| inc0 | by..welll..overriding command | 23:36 |
| kfox1111 | yeah. just a lot more manual | 23:36 |
| sbezverk | kfox1111: in some cases I used debug pod side car | 23:36 |
| sbezverk | worked really well for me | 23:37 |
| kfox1111 | yeah. | 23:37 |
| kfox1111 | we should ask for the same feature for helm. | 23:37 |
| kfox1111 | generally useful for all helm package, not just kolla-kubernetes. | 23:37 |
| sbezverk | kfox1111: just another condition and nothing really required from helm | 23:37 |
| kfox1111 | sbezverk: you shouldn't have to put in contitionals on every chart to debug it. | 23:38 |
| sbezverk | kfox1111: similar concept to entrypoint container | 23:38 |
| kfox1111 | everyt chart in the world could use the feature. | 23:38 |
| kfox1111 | would be silly to instrement every chart to gain it. :/ | 23:38 |
| sbezverk | kfox1111: well, it could be per service | 23:38 |
| kfox1111 | like, all of kubernetes/charts could benifit, for example. | 23:38 |
| *** alop has quit IRC | 23:39 | |
| sbezverk | kfox1111: well, macro is kind of does it, no? just include in every chart adn you are good to go | 23:39 |
| kfox1111 | its trivial to implement in helm's cli. its a huge amount of work to implement it in every chart in the world. | 23:39 |
| kfox1111 | and murphy's law says, the chart you want to debug will be the one without a manual macro. ;) | 23:40 |
| sbezverk | kfox1111: that is 100% true | 23:41 |
| sbezverk | :) | 23:41 |
| kfox1111 | I feel soo sorry for murphy... he's the ultimate messenger. :) | 23:42 |
| *** signed8bit is now known as signed8bit_Zzz | 23:42 | |
| *** esharao has joined #openstack-kolla | 23:44 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Simple configmap to raw kubectl https://review.openstack.org/449262 | 23:45 |
| kfox1111 | https://review.openstack.org/#/c/446274/ is ready for reviews. | 23:47 |
| *** signed8bit_Zzz is now known as signed8bit | 23:49 | |
| kfox1111 | interesting... | 23:51 |
| kfox1111 | https://fosdem.org/2017/schedule/event/kubeoperators/ | 23:51 |
| kfox1111 | havent watched it yet. | 23:51 |
| sbezverk | inc0: ping | 23:53 |
| sbezverk | kfox1111: it looks like one of causes of the issue I see with ubuntu is the version of libvirtd | 23:54 |
| sbezverk | 4.0.0 containers use version 1.3.1 but the vm we get uses 2.5.0 | 23:55 |
| kfox1111 | yeah? did it get upgraded in 4+? | 23:55 |
| kfox1111 | oh. on the host? | 23:55 |
| sbezverk | right | 23:55 |
| kfox1111 | ah. | 23:55 |
| sbezverk | so it seems 4.0.0 has older version | 23:55 |
| kfox1111 | weird. | 23:56 |
| sbezverk | but on centos side these versions are more or less close | 23:56 |
| sbezverk | I mean the container one and the host one | 23:56 |
| sbezverk | I was hoping inc0 could explain how libvirtd version for a container get selected. | 23:56 |
| sbezverk | I mean why 4.0.0 uses 1.3.1 and not 2.5.0 | 23:57 |
| sdake | sbezverk inc0 and Jeffrey4l know quite a bit about that | 23:57 |
| sdake | sbezverk i wasn't really involved inthat, but my understanding is 1.3.1 was the only versio nthat worked in debuntian | 23:57 |
| sbezverk | hm that is a strange statement, since infra provides vm with 2.5.0 | 23:58 |
| sbezverk | and it seems fairly stable | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!