| *** caoyuan has joined #openstack-kolla | 00:01 | |
| openstackgerrit | Monty Taylor proposed openstack/kolla-ansible master: Add clouds.yaml file and use it https://review.openstack.org/452005 | 00:02 |
|---|---|---|
| kfox1111 | hmm.. is that right? .... 172.17.0.2:53 | 00:05 |
| kfox1111 | seems out of range... | 00:06 |
| kfox1111 | yeah, that sees wrong.. | 00:08 |
| kfox1111 | I bet the other host cant get to that. | 00:08 |
| kfox1111 | darn... I bet thats due to the workaround. | 00:09 |
| *** shashank_t_ has quit IRC | 00:10 | |
| sbezverk | kfox1111: where do you see it? | 00:10 |
| sbezverk | kfox1111 | 00:10 |
| kfox1111 | http://logs.openstack.org/91/451391/49/check/gate-kolla-kubernetes-deploy-centos-binary-2-ceph-multi-nv/d2b6aa7/logs/pods.yaml | 00:10 |
| sbezverk | kubectl get nodes -n kube-system | 00:10 |
| sbezverk | NAME STATUS AGE VERSION | 00:10 |
| sbezverk | kube-ubuntu Ready 3h v1.6.0 | 00:10 |
| sbezverk | kube-ubuntu-2 Ready 2m v1.6.0 | 00:10 |
| kfox1111 | grep for podI | 00:10 |
| kfox1111 | grep for podIP | 00:10 |
| *** shashank_t_ has joined #openstack-kolla | 00:10 | |
| sbezverk | kfox1111: hm, how come it pings then? | 00:11 |
| kfox1111 | kube-dns is the one with the bad ip. | 00:12 |
| kfox1111 | its hanging off docker0 I think. | 00:12 |
| sbezverk | wait it is dockers ip not? | 00:12 |
| kfox1111 | yeah. docker0 wouild be 172.17.0.1 | 00:12 |
| kfox1111 | so it works locally, but not across the network. | 00:12 |
| sbezverk | I see the same ip on my local setup | 00:14 |
| kfox1111 | maybe because it comes up before cni. | 00:14 |
| kfox1111 | can you kill the pod after canal starts? | 00:14 |
| kfox1111 | it might get a proper ip then. | 00:14 |
| *** shashank_t_ has quit IRC | 00:15 | |
| sbezverk | kfox1111: sure I can start | 00:15 |
| sbezverk | kfox1111: still the same | 00:16 |
| sbezverk | ip | 00:16 |
| kfox1111 | very weird... | 00:16 |
| kfox1111 | its not net=host right? | 00:17 |
| sbezverk | we need to provide fixed ip to the container, maybe then it will work | 00:17 |
| kfox1111 | I'm thinking its avoiding cni... | 00:17 |
| kfox1111 | if you kubectl run a pod that lands on the first node, does it get a 172.17 address? | 00:18 |
| kfox1111 | its not net=host. | 00:21 |
| kfox1111 | we should see if we can undo the workaround. | 00:22 |
| sbezverk | kfox1111: yeah it gets ip from docker's range | 00:22 |
| kfox1111 | can you reenable it on your primary, systemctl daemon-reload; systemctl restart kubelet; | 00:23 |
| kfox1111 | and kill the kube-dns pod? | 00:23 |
| kfox1111 | maybe the problem was kube-dns was sticking around. | 00:23 |
| sbezverk | kfox1111: I mean I can try but I think dns pod is not relevant | 00:24 |
| sbezverk | every pod gets wrong ip address | 00:24 |
| sbezverk | I brought up centos pod | 00:24 |
| sbezverk | and it got ip from docker range | 00:24 |
| kfox1111 | interesting... | 00:24 |
| kfox1111 | podIP: 172.17.0.3 on the other test-dns pod. | 00:25 |
| kfox1111 | so that would make sense it would work. | 00:25 |
| kfox1111 | sbezverk: which host did it land on? | 00:25 |
| *** adrian_otto has quit IRC | 00:26 | |
| sbezverk | on 2nd node | 00:27 |
| kfox1111 | ok. so we'd have to reenable the cni plugin on both nodes... | 00:28 |
| kfox1111 | or change the workaround to only apply on the first node, | 00:28 |
| kfox1111 | then we only need to clean up the first node. | 00:28 |
| sbezverk | kfox1111: ok I will try to revert workaround on "master: for now | 00:28 |
| kfox1111 | yeah, confirmed. I'm seeing docker0 ip's on both nodes too. | 00:28 |
| *** zhurong has joined #openstack-kolla | 00:29 | |
| *** caoyuan has quit IRC | 00:32 | |
| sbezverk | kfox1111: reverting workaround fixed | 00:34 |
| sbezverk | new pod got right ip address | 00:34 |
| *** MasterOfBugs has quit IRC | 00:35 | |
| sbezverk | kfox1111: deleting kube-dns pod and then re-creating it, fixes it as well | 00:36 |
| sbezverk | kfox1111: so we have to revert after all init/join are completed | 00:37 |
| *** sayantan_ has joined #openstack-kolla | 00:40 | |
| *** tovin07_ has joined #openstack-kolla | 00:44 | |
| kfox1111 | easiest would be to put a conditional around the network removal so it only happens on the master. | 00:44 |
| kfox1111 | then we only need to fix it there. | 00:45 |
| sbezverk | kfox1111: not sure if it is on the master | 00:45 |
| masber | hi, need some clarification, has anyone here deployed openstack control plane using kolla on a vmware environment? because I am trying to do it but my router namespace is not able to ping outside and I am wondering if there is an issue with vmware | 00:45 |
| sbezverk | i did it on both | 00:45 |
| kfox1111 | sbezverk: the workaround is only needed to get kubeadm to boot. | 00:46 |
| kfox1111 | I think. | 00:46 |
| *** yangyape_ has joined #openstack-kolla | 00:46 | |
| kfox1111 | if we start the slave with the cni driver flag still enabled, Ithink it will be fine. | 00:46 |
| sbezverk | kfox1111: ont exactly, it chnages kubelet behavior | 00:46 |
| sbezverk | it might also impact join | 00:47 |
| kfox1111 | pretty sure it will work. but don't know for sure. | 00:47 |
| sbezverk | kfox1111: if you want to take a stab at it on the latest version of ps, go for it.. | 00:48 |
| *** KTJ has joined #openstack-kolla | 00:48 | |
| sbezverk | I will try to clean up slave and try it locally | 00:49 |
| *** yangyapeng has quit IRC | 00:49 | |
| *** KTJ has quit IRC | 00:49 | |
| *** cuongnv has joined #openstack-kolla | 00:49 | |
| *** KTJ has joined #openstack-kolla | 00:50 | |
| *** yangyape_ has quit IRC | 00:50 | |
| kfox1111 | kk | 00:51 |
| *** yingjun has joined #openstack-kolla | 00:54 | |
| sbezverk | kfox1111: it looks like it worked | 00:56 |
| sbezverk | I did kubeadm reset then reboot | 00:56 |
| kfox1111 | cool. | 00:57 |
| *** yuanying has quit IRC | 00:58 | |
| *** eanylin has quit IRC | 01:02 | |
| *** KTJ has quit IRC | 01:03 | |
| kfox1111 | almost done... | 01:04 |
| kfox1111 | close. | 01:07 |
| kfox1111 | sbezverk: can you please find a label selector for kube-dns? | 01:07 |
| kfox1111 | to identify the pod for deletion. | 01:07 |
| sbezverk | kfox1111: one sec | 01:07 |
| sbezverk | k8s-app: kube-dns | 01:08 |
| kfox1111 | kubectl get pods -l k8s-app=kube-dns -n kube-system -o json | jq .items[].metadata.name ? | 01:09 |
| *** yuanying has joined #openstack-kolla | 01:10 | |
| sbezverk | yep, but is quoted | 01:10 |
| kfox1111 | kubectl get pods -l k8s-app=kube-dns -n kube-system -o json | jq -r .items[].metadata.name ? | 01:10 |
| sbezverk | "kube-dns-1080222536-zk7th" | 01:10 |
| *** caoyuan has joined #openstack-kolla | 01:11 | |
| kfox1111 | lets see if it will be recreated on restart. | 01:12 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 upgrade https://review.openstack.org/451391 | 01:14 |
| *** yangyapeng has joined #openstack-kolla | 01:15 | |
| sbezverk | kfox1111: where do you save original network settings? | 01:18 |
| kfox1111 | the file itself. :) | 01:19 |
| kfox1111 | I just tag a # to the front of the line, then remove it. :) | 01:19 |
| sbezverk | kfox1111: I see, cool | 01:20 |
| openstackgerrit | caoyuan proposed openstack/kolla master: Install zun ui into horizon image https://review.openstack.org/452019 | 01:23 |
| *** zhubingbing_ has joined #openstack-kolla | 01:24 | |
| *** yangyapeng has quit IRC | 01:29 | |
| *** yangyapeng has joined #openstack-kolla | 01:30 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible master: Enable zun ui when zaqar enabled https://review.openstack.org/452020 | 01:30 |
| *** eanylin has joined #openstack-kolla | 01:33 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible master: Enable zun ui when zaqar enabled https://review.openstack.org/452020 | 01:34 |
| *** yangyapeng has quit IRC | 01:35 | |
| *** ubuntu__ has joined #openstack-kolla | 01:38 | |
| *** eanylin has quit IRC | 01:40 | |
| *** manheim has joined #openstack-kolla | 01:43 | |
| Jeffrey4l | sup mnaser | 01:44 |
| Jeffrey4l | sdake, re k8s doc change gate, i know how. i will push a fix today. | 01:44 |
| *** lucasxu has joined #openstack-kolla | 01:45 | |
| kfox1111 | sbezverk: success | 01:48 |
| *** manheim has quit IRC | 01:48 | |
| kfox1111 | some random failures still, but multinode's passing. | 01:48 |
| *** oanson has quit IRC | 01:55 | |
| *** oanson has joined #openstack-kolla | 01:56 | |
| *** yangyapeng has joined #openstack-kolla | 01:56 | |
| *** dixiaoli has joined #openstack-kolla | 01:58 | |
| *** yangyapeng has quit IRC | 02:07 | |
| *** eaguilar has quit IRC | 02:08 | |
| duonghq | sdake, ping | 02:10 |
| *** lucasxu has quit IRC | 02:10 | |
| *** lucasxu has joined #openstack-kolla | 02:11 | |
| *** yangyapeng has joined #openstack-kolla | 02:13 | |
| sbezverk | kfox1111: congrats!!! | 02:13 |
| *** yangyapeng has quit IRC | 02:14 | |
| kfox1111 | sbezverk: you too!! :) | 02:15 |
| openstackgerrit | Surya Prakash (spsurya) proposed openstack/kolla-kubernetes master: README.txt character improvement https://review.openstack.org/451654 | 02:18 |
| *** lucasxu has quit IRC | 02:19 | |
| *** goldyfruit has joined #openstack-kolla | 02:29 | |
| openstackgerrit | Surya Prakash (spsurya) proposed openstack/kolla-kubernetes master: README.rst character improvement https://review.openstack.org/451654 | 02:30 |
| *** jrobinson has quit IRC | 02:30 | |
| *** caoyuan has quit IRC | 02:33 | |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla master: Spec - Zero-downtime upgrade in Kolla https://review.openstack.org/409598 | 02:38 |
| *** yangyapeng has joined #openstack-kolla | 02:43 | |
| *** caoyuan has joined #openstack-kolla | 02:57 | |
| *** yangyapeng has quit IRC | 02:57 | |
| *** yangyapeng has joined #openstack-kolla | 03:02 | |
| sdake | duonghq sup | 03:07 |
| duonghq | sdake, why we need unsigned package for k8s 1.5.4 in centos? the official ones are removed? | 03:08 |
| *** jrobinson has joined #openstack-kolla | 03:09 | |
| *** yuanying has quit IRC | 03:11 | |
| *** britthouser has quit IRC | 03:14 | |
| *** yuanying has joined #openstack-kolla | 03:14 | |
| *** britthouser has joined #openstack-kolla | 03:14 | |
| *** shashank_t_ has joined #openstack-kolla | 03:16 | |
| *** yangyape_ has joined #openstack-kolla | 03:16 | |
| *** yangyapeng has quit IRC | 03:17 | |
| *** yangyapeng has joined #openstack-kolla | 03:17 | |
| *** yangyapeng has quit IRC | 03:20 | |
| sdake | Jeffrey4l thanks :) | 03:20 |
| *** yangyape_ has quit IRC | 03:21 | |
| *** caoyuan has quit IRC | 03:22 | |
| *** manheim has joined #openstack-kolla | 03:30 | |
| *** prameswar has joined #openstack-kolla | 03:30 | |
| *** manheim has quit IRC | 03:35 | |
| *** caoyuan has joined #openstack-kolla | 03:40 | |
| *** lamt has joined #openstack-kolla | 03:42 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible master: Fix the check.yml of Zun does not exist https://review.openstack.org/452038 | 03:43 |
| *** dixiaoli has quit IRC | 03:48 | |
| *** dixiaoli has joined #openstack-kolla | 03:52 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 upgrade https://review.openstack.org/451391 | 03:58 |
| sbezverk | sdake: sdake | 03:59 |
| sbezverk | sdake: nv http://logs.openstack.org/91/451391/50/check/gate-kolla-kubernetes-deploy-centos-binary-2-ceph-multi-nv/de00799/ | 04:00 |
| *** dixiaoli has quit IRC | 04:00 | |
| *** britthouser has quit IRC | 04:06 | |
| *** britthouser has joined #openstack-kolla | 04:06 | |
| *** zhurong has quit IRC | 04:09 | |
| *** n0isyn0i1e has joined #openstack-kolla | 04:10 | |
| *** goldyfruit has quit IRC | 04:11 | |
| *** ubuntu__ has quit IRC | 04:12 | |
| *** ubuntu__ has joined #openstack-kolla | 04:13 | |
| *** n0isyn0ise has quit IRC | 04:13 | |
| *** eanylin has joined #openstack-kolla | 04:14 | |
| *** gkadam has joined #openstack-kolla | 04:17 | |
| *** japestinho has quit IRC | 04:20 | |
| *** dixiaoli has joined #openstack-kolla | 04:24 | |
| *** yangyapeng has joined #openstack-kolla | 04:24 | |
| *** dixiaoli has quit IRC | 04:25 | |
| openstackgerrit | zhubingbing proposed openstack/kolla-kubernetes master: Fix problem with kollakube res create configmap https://review.openstack.org/439740 | 04:28 |
| *** zhurong has joined #openstack-kolla | 04:30 | |
| *** prameswar has quit IRC | 04:32 | |
| *** duonghq has quit IRC | 04:40 | |
| *** skramaja has joined #openstack-kolla | 04:43 | |
| *** bmace has quit IRC | 04:44 | |
| *** bmace has joined #openstack-kolla | 04:44 | |
| *** yangyapeng has quit IRC | 04:55 | |
| *** yangyapeng has joined #openstack-kolla | 04:56 | |
| openstackgerrit | zhubingbing proposed openstack/kolla-kubernetes master: Fix problem with kollakube res create configmap https://review.openstack.org/439740 | 04:58 |
| *** yangyapeng has quit IRC | 05:00 | |
| *** lamt has quit IRC | 05:19 | |
| *** targon has joined #openstack-kolla | 05:23 | |
| *** jrobinson has quit IRC | 05:23 | |
| *** jrobinson has joined #openstack-kolla | 05:28 | |
| *** dixiaoli has joined #openstack-kolla | 05:35 | |
| *** zhurong has quit IRC | 05:41 | |
| *** lamt has joined #openstack-kolla | 05:42 | |
| *** dixiaoli has quit IRC | 05:43 | |
| *** yangyapeng has joined #openstack-kolla | 05:52 | |
| *** prameswar has joined #openstack-kolla | 05:52 | |
| *** adrian_otto has joined #openstack-kolla | 05:52 | |
| *** jaosorior has joined #openstack-kolla | 05:56 | |
| *** dixiaoli has joined #openstack-kolla | 05:59 | |
| *** dixiaoli has quit IRC | 06:00 | |
| *** zhurong has joined #openstack-kolla | 06:05 | |
| openstackgerrit | jimmygc proposed openstack/kolla-ansible master: Add VMware NSXV support to neutron https://review.openstack.org/450092 | 06:05 |
| *** duonghq has joined #openstack-kolla | 06:07 | |
| *** yangyapeng has quit IRC | 06:08 | |
| *** yangyapeng has joined #openstack-kolla | 06:08 | |
| openstackgerrit | Surya Prakash (spsurya) proposed openstack/kolla-kubernetes master: Deployment Guide Documentation https://review.openstack.org/447356 | 06:13 |
| *** yangyapeng has quit IRC | 06:13 | |
| *** dixiaoli has joined #openstack-kolla | 06:14 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible master: Update "always_run" to "check_mode" https://review.openstack.org/452056 | 06:23 |
| *** yangyapeng has joined #openstack-kolla | 06:26 | |
| *** Serlex has joined #openstack-kolla | 06:29 | |
| *** yangyapeng has quit IRC | 06:31 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible master: Update "always_run" to "check_mode" https://review.openstack.org/452056 | 06:40 |
| *** jmccarthy has joined #openstack-kolla | 06:43 | |
| *** satyar has joined #openstack-kolla | 06:45 | |
| openstackgerrit | jimmygc proposed openstack/kolla-ansible master: Add Glance Swift backend support https://review.openstack.org/452059 | 06:47 |
| duonghq | Jeffrey4l, are you there? | 06:50 |
| duonghq | about this ps: https://review.openstack.org/#/c/452056/2 | 06:51 |
| openstackgerrit | shaofeng cheng proposed openstack/kolla-ansible master: Fix secure_proxy_ssl_header option in nova https://review.openstack.org/452062 | 06:51 |
| Jeffrey4l | duonghq, yep. | 06:51 |
| duonghq | is it right time to do the migration? | 06:51 |
| duonghq | we revert it once it the past | 06:51 |
| Jeffrey4l | duonghq, before doing this. we need bump the min ansible version to 2.1 or 2.2. | 06:51 |
| duonghq | *reverted | 06:51 |
| Jeffrey4l | i think bump the ansible version is OK. | 06:52 |
| Jeffrey4l | in this cycle. | 06:52 |
| Jeffrey4l | but this patch shouldn't be merged until bump ansible version is merged. | 06:52 |
| *** jrobinson has quit IRC | 06:52 | |
| duonghq | Jeffrey4l, got it, can you do the dump when you feel convenient? | 06:52 |
| duonghq | *bmp | 06:52 |
| duonghq | *bump | 06:53 |
| Jeffrey4l | np. | 06:53 |
| Jeffrey4l | btw, i -1 for that patch. | 06:53 |
| duonghq | Jeffrey4l, ok | 06:54 |
| openstackgerrit | Surya Prakash (spsurya) proposed openstack/kolla-kubernetes master: Deployment Guide Documentation https://review.openstack.org/447356 | 06:57 |
| *** imcsk8 has joined #openstack-kolla | 06:58 | |
| *** imcsk8_ has quit IRC | 06:58 | |
| *** yangyapeng has joined #openstack-kolla | 07:05 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible master: Bump min Ansible version to 2.2.0 https://review.openstack.org/452067 | 07:05 |
| *** japestinho has joined #openstack-kolla | 07:07 | |
| *** jimmygc has joined #openstack-kolla | 07:15 | |
| *** matrohon has joined #openstack-kolla | 07:16 | |
| *** yingjun has quit IRC | 07:18 | |
| *** lamt has quit IRC | 07:20 | |
| *** pcaruana has joined #openstack-kolla | 07:27 | |
| *** sayantan_ has quit IRC | 07:28 | |
| duonghq | Jeffrey4l, why we do not have ansible in requirement | 07:28 |
| Jeffrey4l | duonghq, hrm.. the items in requirements.txt should be exist in requirements project first, iirc. | 07:29 |
| *** dixiaoli has quit IRC | 07:29 | |
| Jeffrey4l | otherwise, the gate jobs will be red. | 07:29 |
| Jeffrey4l | But you can add it to have a try. | 07:30 |
| *** caoyuan_ has joined #openstack-kolla | 07:30 | |
| *** prameswar has quit IRC | 07:30 | |
| *** mgoddard has joined #openstack-kolla | 07:30 | |
| duonghq | Jeffrey4l, ok | 07:31 |
| *** n0isyn0i1e is now known as n0isyn0ise | 07:31 | |
| *** dixiaoli has joined #openstack-kolla | 07:32 | |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla-ansible master: Add ansible to requirement https://review.openstack.org/452075 | 07:33 |
| *** caoyuan has quit IRC | 07:33 | |
| *** shashank_t_ has quit IRC | 07:33 | |
| *** shashank_t_ has joined #openstack-kolla | 07:34 | |
| openstackgerrit | Surya Prakash (spsurya) proposed openstack/kolla master: Fix oslo_debug_helper https://review.openstack.org/444398 | 07:37 |
| *** Deys has joined #openstack-kolla | 07:38 | |
| *** shashank_t_ has quit IRC | 07:38 | |
| openstackgerrit | Surya Prakash (spsurya) proposed openstack/kolla master: Fix oslo_debug_helper not working https://review.openstack.org/444398 | 07:40 |
| *** yangyapeng has quit IRC | 07:41 | |
| *** yangyapeng has joined #openstack-kolla | 07:41 | |
| -openstackstatus- NOTICE: Jobs in gate are failing with POST_FAILURE. Infra roots are investigating | 07:43 | |
| *** ChanServ changes topic to "Jobs in gate are failing with POST_FAILURE. Infra roots are investigating" | 07:43 | |
| *** sayantan_ has joined #openstack-kolla | 07:48 | |
| *** egonzalez has joined #openstack-kolla | 07:48 | |
| *** matrohon has quit IRC | 07:55 | |
| *** matrohon has joined #openstack-kolla | 07:56 | |
| *** caowei has joined #openstack-kolla | 08:01 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible master: Enable zun ui when zun enabled https://review.openstack.org/452020 | 08:01 |
| *** gfidente has joined #openstack-kolla | 08:02 | |
| *** dixiaoli has quit IRC | 08:05 | |
| *** dixiaoli has joined #openstack-kolla | 08:05 | |
| *** athomas has joined #openstack-kolla | 08:06 | |
| *** manheim has joined #openstack-kolla | 08:07 | |
| *** manheim has quit IRC | 08:12 | |
| *** sayantan_ has quit IRC | 08:15 | |
| openstackgerrit | jimmygc proposed openstack/kolla-ansible master: Make glance filesystem_store_datadir configurable. https://review.openstack.org/452094 | 08:20 |
| *** jimmygc_ has joined #openstack-kolla | 08:22 | |
| -openstackstatus- NOTICE: logs.openstack.org has corrupted disks, it's being repaired. Please avoid rechecking until this is fixed | 08:23 | |
| *** ChanServ changes topic to "logs.openstack.org has corrupted disks, it's being repaired. Please avoid rechecking until this is fixed" | 08:23 | |
| *** jimmygc has quit IRC | 08:24 | |
| *** shardy has joined #openstack-kolla | 08:26 | |
| *** papacz has joined #openstack-kolla | 08:29 | |
| openstackgerrit | James McCarthy proposed openstack/kolla-ansible master: Change default permissions in jinja templates. https://review.openstack.org/445797 | 08:34 |
| *** shashank_t_ has joined #openstack-kolla | 08:36 | |
| *** shardy has quit IRC | 08:37 | |
| *** eaguilar has joined #openstack-kolla | 08:38 | |
| *** shardy has joined #openstack-kolla | 08:38 | |
| *** sayantan_ has joined #openstack-kolla | 08:44 | |
| *** dixiaoli has quit IRC | 08:45 | |
| *** eaguilar has quit IRC | 08:47 | |
| *** shashank_t_ has quit IRC | 08:48 | |
| *** sayantan_ has quit IRC | 08:49 | |
| *** ipsecguy has quit IRC | 08:51 | |
| *** ipsecguy has joined #openstack-kolla | 08:51 | |
| *** matrohon has quit IRC | 08:51 | |
| *** matrohon has joined #openstack-kolla | 08:52 | |
| *** pbandark has joined #openstack-kolla | 08:59 | |
| *** matrohon has quit IRC | 08:59 | |
| *** manheim has joined #openstack-kolla | 09:03 | |
| *** yangyape_ has joined #openstack-kolla | 09:04 | |
| *** blallau has joined #openstack-kolla | 09:05 | |
| *** dixiaoli has joined #openstack-kolla | 09:05 | |
| *** yangyapeng has quit IRC | 09:07 | |
| satyar | hi inc0 | 09:12 |
| satyar | how are you? | 09:12 |
| *** matrohon has joined #openstack-kolla | 09:17 | |
| *** varto has joined #openstack-kolla | 09:21 | |
| *** rmart04 has joined #openstack-kolla | 09:21 | |
| *** caoyuan_ has quit IRC | 09:25 | |
| *** caoyuan has joined #openstack-kolla | 09:26 | |
| *** yangyape_ has quit IRC | 09:26 | |
| *** matrohon_ has joined #openstack-kolla | 09:28 | |
| *** matrohon has quit IRC | 09:30 | |
| *** caoyuan_ has joined #openstack-kolla | 09:30 | |
| hrw | morning | 09:30 |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla-ansible master: Update Keystone haproxy config to balance based on source ip https://review.openstack.org/451358 | 09:31 |
| *** yangyapeng has joined #openstack-kolla | 09:31 | |
| *** mgoddard has quit IRC | 09:32 | |
| *** caoyuan has quit IRC | 09:33 | |
| *** zhuzeyu has quit IRC | 09:37 | |
| sdake | morning folks | 09:38 |
| *** sayantan_ has joined #openstack-kolla | 09:39 | |
| sdake | egonzalez since you modified the review directly you can remove your-1 | 09:39 |
| sdake | for https://review.openstack.org/#/c/346455/ | 09:40 |
| egonzalez | sdake, done | 09:40 |
| *** sayantan_ has quit IRC | 09:43 | |
| *** rmart04 has quit IRC | 09:48 | |
| *** adrian_otto has quit IRC | 09:53 | |
| *** yangyapeng has quit IRC | 09:54 | |
| *** yangyapeng has joined #openstack-kolla | 09:54 | |
| *** mgoddard has joined #openstack-kolla | 10:00 | |
| *** athomas has quit IRC | 10:01 | |
| *** duonghq has quit IRC | 10:04 | |
| varto | Having issue with installing multinode kolla cinder block storage working just fine (creates volumes), but when starting vm getting error can't connect to any Iscsi portal? | 10:07 |
| *** matrohon_ has quit IRC | 10:08 | |
| *** matrohon_ has joined #openstack-kolla | 10:08 | |
| varto | redeploying currently and double checking nova-compute configs, but just haven't found way to fix this. | 10:09 |
| *** athomas has joined #openstack-kolla | 10:09 | |
| *** athomas has quit IRC | 10:09 | |
| *** athomas has joined #openstack-kolla | 10:09 | |
| *** caoyuan_ has quit IRC | 10:11 | |
| openstackgerrit | shaofeng cheng proposed openstack/kolla-ansible master: Add VMware DataStore support to cinder https://review.openstack.org/452131 | 10:13 |
| *** zhubingbing_ has quit IRC | 10:13 | |
| egonzalez | varto, can you please check if iscsi and tgtd containers are running and bind to the correct IP address? share config, globals and error logs | 10:17 |
| varto | tgtd and iscsi was running, but I'll check those again after redeploy. | 10:18 |
| *** zhurong has quit IRC | 10:18 | |
| *** tovin07_ has quit IRC | 10:22 | |
| *** sambetts|afk is now known as sambetts | 10:25 | |
| *** satyar has quit IRC | 10:28 | |
| *** caoyuan has joined #openstack-kolla | 10:30 | |
| *** pbourke has quit IRC | 10:30 | |
| *** pbourke has joined #openstack-kolla | 10:32 | |
| *** sayantan_ has joined #openstack-kolla | 10:33 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla master: WIP: Add nginx image https://review.openstack.org/452134 | 10:33 |
| *** kolla-slack1 has quit IRC | 10:36 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla master: WIP: Add nginx image https://review.openstack.org/452134 | 10:36 |
| *** kolla-slack has joined #openstack-kolla | 10:36 | |
| *** varto has quit IRC | 10:37 | |
| *** sayantan_ has quit IRC | 10:37 | |
| *** matrohon_ has quit IRC | 10:38 | |
| *** cuongnv has quit IRC | 10:40 | |
| *** zhurong has joined #openstack-kolla | 10:41 | |
| *** dixiaoli has quit IRC | 10:49 | |
| *** dixiaoli has joined #openstack-kolla | 10:51 | |
| openstackgerrit | Bertrand Lallau proposed openstack/kolla-ansible master: Karbor services restarting issue on conf update https://review.openstack.org/451735 | 10:52 |
| *** manheim has quit IRC | 10:52 | |
| openstackgerrit | Bertrand Lallau proposed openstack/kolla-ansible master: Trove services restarting issue on conf update https://review.openstack.org/451720 | 10:56 |
| *** rhallisey has joined #openstack-kolla | 10:59 | |
| *** manheim has joined #openstack-kolla | 11:08 | |
| *** shardy is now known as shardy_lunch | 11:14 | |
| targon | Hi! Quick question. I installed openstack 4.0.0. (all-in-one) to Ubuntu 16.10 running in KVM. Othervise ok, but all static content missing when logged into Horizon. Only icon I see is Openstack icon in top left corner. I can start cirros instances there etc. but dasboard layout is total mess without static content (and propably scripts). | 11:14 |
| targon | Any idea what I have missed? I followed instructions in https://docs.openstack.org/project-deploy-guide/kolla-ansible/ocata/quickstart.html . When tried 3.0.2 release i got icons, but there were other problems. Which version I should install? | 11:14 |
| hrw | http://people.linaro.org/~marcin.juszkiewicz/kolla/ | 11:17 |
| hrw | nice table with effects of builds for different architectures | 11:18 |
| *** jtriley has quit IRC | 11:19 | |
| hrw | what do you think? egonzalez sdake pbourke | 11:22 |
| sdake | hrw looks cool | 11:23 |
| sdake | hrw ought to post to te mailing list for wider audience | 11:23 |
| hrw | sure | 11:23 |
| sdake | hrw can i ask - are ou getting reviews on your work in sufficient time? | 11:23 |
| hrw | sdake: I took time two days ago and split whole patchset into separate ones + one main one. so far reviews are quite ok | 11:24 |
| hrw | sdake: I think that random gates failures are bigger issue than review speed | 11:25 |
| *** sayantan_ has joined #openstack-kolla | 11:27 | |
| *** sayantan_ has quit IRC | 11:31 | |
| *** matrohon has joined #openstack-kolla | 11:38 | |
| *** matrohon has quit IRC | 11:41 | |
| *** jimmygc_ has quit IRC | 11:41 | |
| *** srwilkers has joined #openstack-kolla | 11:42 | |
| *** jimmygc has joined #openstack-kolla | 11:45 | |
| *** yangyapeng has quit IRC | 11:45 | |
| *** yangyapeng has joined #openstack-kolla | 11:46 | |
| *** skramaja_ has joined #openstack-kolla | 11:50 | |
| *** yangyapeng has quit IRC | 11:51 | |
| *** skramaja has quit IRC | 11:52 | |
| openstackgerrit | jimmygc proposed openstack/kolla-ansible master: Add Glance Swift backend support https://review.openstack.org/452059 | 11:53 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Fix Haproxy reconfigure https://review.openstack.org/452166 | 11:53 |
| *** dixiaoli has quit IRC | 11:54 | |
| *** magicboiz has quit IRC | 11:55 | |
| *** bjolo has joined #openstack-kolla | 11:59 | |
| *** zhurong has quit IRC | 11:59 | |
| sdake | hrw ya - no solution for those random gate failures yet | 11:59 |
| sdake | hrw annoying as they are | 11:59 |
| hrw | yep | 12:00 |
| *** schwicht has quit IRC | 12:00 | |
| sdake | Jeffrey4l kfox1111 sort of had an idea about uploading kubernetes rpms to tarballs.oo - and I had a riff on that idea which was to upload all RPM dependencies (from a squid cache possibly?) used for each build to tarballs.oo | 12:01 |
| sdake | this would permit the ability to completely reproduce the build from what was gated | 12:01 |
| *** shardy_lunch is now known as shardy | 12:03 | |
| *** goldyfruit has joined #openstack-kolla | 12:04 | |
| *** goldyfruit has quit IRC | 12:04 | |
| openstackgerrit | shaofeng cheng proposed openstack/kolla-ansible master: Add VMware DataStore support to glance https://review.openstack.org/452176 | 12:08 |
| mnaser | morning o/ | 12:09 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: WIP: Split keepalived own role https://review.openstack.org/452177 | 12:11 |
| Jeffrey4l | sdake, for squid solution is hard. squid can not save every. imo. | 12:11 |
| Jeffrey4l | everything* | 12:11 |
| sdake | Jeffrey4l well you get hte idea | 12:11 |
| sdake | squid was just something that popped out as an implementation idea | 12:12 |
| Jeffrey4l | why we need upload rpm to tarballs? | 12:12 |
| sdake | people comain the build is not reproducible | 12:12 |
| sdake | what yo ubuild today may not work tomorrow | 12:12 |
| Jeffrey4l | i do not think this is caused by *rpm* | 12:12 |
| *** rwellum has joined #openstack-kolla | 12:12 | |
| sdake | its caused by changing dependencies in the host os | 12:12 |
| sdake | rather container os | 12:13 |
| Jeffrey4l | for example? if so devstack will be failed more often. | 12:13 |
| sdake | probably the pip stuff needs to be uploaded as well for source installs? | 12:13 |
| *** dciabrin has quit IRC | 12:13 | |
| sdake | what do you think it is caused by | 12:13 |
| Jeffrey4l | sdake, there is a mirror in infra env. pip/centos repo/ ubuntu etc | 12:14 |
| Jeffrey4l | i need to know which issue we are talking about. | 12:14 |
| Jeffrey4l | but i am sure, lots of gate failure is caused by rpm dependency. | 12:14 |
| Jeffrey4l | is not* ( sorry) | 12:15 |
| sdake | Jeffrey4l i lost a msg there could you repeat | 12:15 |
| Jeffrey4l | i need to know which issue we are talking about. but i am sure, lots of gate failure are not caused by rpm dependency. | 12:16 |
| sdake | Jeffrey4l just a suggestion on how to generate reproducible builds | 12:16 |
| sdake | oh right - this isnt' about gate failures - completely orthogonal | 12:16 |
| sdake | i htink in terms of priorities making gate reliable is higher priority | 12:17 |
| Jeffrey4l | we need to know what is changed between gate and local env. and which change is necessary. | 12:17 |
| sdake | onlly something i want to pass along from a discussion i had yesterday or the day before | 12:17 |
| Jeffrey4l | rpm dependency / repo shouldn't be the necessary. | 12:18 |
| sdake | problem: we are not offering a way for operators to reproduce the builds we generate in the gate. We cannot reproduce the builds ourselves as the entire distro chain and all binaries used to build containers change constantly | 12:18 |
| Jeffrey4l | for making the gate more reliable, we'd better capture more and more logs/env/variable from gate for analysis. more failed only happen in gate, imo. | 12:19 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Fix designate prechecks https://review.openstack.org/452180 | 12:19 |
| sdake | this isn't about reliability - this is about reproducibility | 12:19 |
| sdake | let me make a simpler example | 12:20 |
| Jeffrey4l | " reproduce the builds we generate in the gate" - hrm, are you not talking the success or failure. are you talking the exact rpm version in the images? | 12:20 |
| sdake | lets say i have a file called dothis.py on my hard disk | 12:20 |
| sdake | i run it | 12:20 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Fix designate prechecks https://review.openstack.org/452180 | 12:20 |
| sdake | then later i edit it | 12:20 |
| sdake | i run it after editing | 12:20 |
| sdake | the old version is long gone | 12:21 |
| sdake | how do i get the results from the old dothis.py? | 12:21 |
| Jeffrey4l | no way if you are not using git. | 12:21 |
| sdake | right | 12:21 |
| sdake | and git does what? | 12:21 |
| Jeffrey4l | back and get the old dothis.py file | 12:22 |
| sdake | right - git stores every version of the dothis.py file | 12:22 |
| sdake | or atleast evey version you care to store | 12:22 |
| sdake | we need the same thing for gate dependencies for tagged builds | 12:22 |
| Jeffrey4l | why? | 12:23 |
| sdake | we can't reproduce our builds | 12:23 |
| sdake | a build that works today may not work in 3 months | 12:23 |
| Jeffrey4l | you can get the same image from the running container or you old registry. | 12:23 |
| sdake | because the RPMS change or DEBs change or xyz changes | 12:23 |
| sdake | need archive of all the build dependencies | 12:23 |
| Jeffrey4l | yep rpms change. | 12:23 |
| *** schwicht has joined #openstack-kolla | 12:23 | |
| Jeffrey4l | you can get the exact docker images. why you wanna build it again? | 12:24 |
| sdake | if rpms change and there is no archive, how do you build an old version? | 12:24 |
| sdake | docker images dont contain the rpms | 12:24 |
| Jeffrey4l | i do not need build an old version. | 12:24 |
| sdake | its a genreal requirement thta keeps popping up | 12:24 |
| Jeffrey4l | first of all, the old images should be saved in docker registry. you can get the old image from there. | 12:24 |
| Jeffrey4l | second, you can get the old images from the running container.( which is ran from the old images) | 12:25 |
| Jeffrey4l | i can get the old images in two ways. why i need build it agains? | 12:25 |
| sdake | what if you wan tto change the container in some way (add a plugin) but keep all other things equal? | 12:26 |
| mnaser | hi guys | 12:26 |
| mnaser | http://paste.openstack.org/show/605048/ and http://paste.openstack.org/show/604927/ | 12:26 |
| sdake | sup mnaser | 12:26 |
| Jeffrey4l | hey mnaser | 12:26 |
| mnaser | images which failed to build | 12:26 |
| mnaser | (so far) | 12:26 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Fix designate prechecks https://review.openstack.org/452180 | 12:26 |
| *** ccesario has quit IRC | 12:26 | |
| mnaser | ive reran the build twice and they failed in both of these scenarios. | 12:26 |
| mnaser | im building ubuntu source now but can i get an ok to push even if those failed? | 12:27 |
| sdake | mnaser binary doesn't have all things paackaged | 12:27 |
| sdake | mnaser some things fail normally | 12:27 |
| sdake | mnaser the logs could be more helpful about this point | 12:27 |
| Jeffrey4l | sdake, hrm, you can build a new set of images to upgrade ;) | 12:27 |
| sdake | mnaser the exceptions list is in the tests dir | 12:27 |
| mnaser | sdake i figured, but when i saw some of them say "error" instead of "matched" .. it was a bit concerning | 12:28 |
| mnaser | okay, i'll double check that | 12:28 |
| mnaser | i guess the exception list when running kolla-build isnt shared | 12:28 |
| Jeffrey4l | mnaser, some images are failed for sure. | 12:28 |
| Jeffrey4l | you may need check test_build.py file. | 12:28 |
| *** signed8bit_Zzz is now known as signed8b_ | 12:28 | |
| sdake | mnaser ya that whole output thing is confusing | 12:28 |
| sdake | it should be more straightforward | 12:28 |
| sdake | This container buildt; X" | 12:28 |
| mnaser | the list should be shared as well, i'll see what we can do to clear that out | 12:28 |
| sdake | This container didn't build because (reason): X" | 12:28 |
| mnaser | Jeffrey4l in other news ive made progress from your patch to run tempest smoke tests | 12:29 |
| mnaser | ubuntu has this weird issue that is failing (but seems to be tempest related) because centos and oraclelinux both run ok | 12:30 |
| Jeffrey4l | mnaser, great. | 12:30 |
| mnaser | id show the logs but logs.openstack.org is not feeling good | 12:30 |
| Jeffrey4l | mnaser, do u have a patch link? | 12:31 |
| mnaser | Jeffrey4l https://review.openstack.org/#/c/402122/ still a wip | 12:31 |
| Jeffrey4l | aha, you re-used mine. let me check. | 12:31 |
| mnaser | oh and i made some tempest changes so we dont have to blacklist checks | 12:32 |
| mnaser | yep :) | 12:32 |
| mnaser | the blacklist file is not used so ill have to remove that | 12:32 |
| Jeffrey4l | nice. | 12:32 |
| *** schwicht has quit IRC | 12:32 | |
| *** schwicht_at_work has joined #openstack-kolla | 12:32 | |
| mnaser | sorry for overchecking most likely but i want ot make sure things are a-ok | 12:33 |
| mnaser | for ubuntu source and binary, does this seem like it's reasonable to push? http://paste.openstack.org/show/605050/ | 12:33 |
| mnaser | just a short "hey this seems like its the right set of images with the right tags" is enough | 12:33 |
| Jeffrey4l | mnaser, let me check the test_build.py file. | 12:33 |
| Jeffrey4l | 1sec | 12:33 |
| mnaser | thank you | 12:34 |
| Jeffrey4l | mnaser, for ubuntu + binary, all failed are expected except watcher. | 12:35 |
| Jeffrey4l | but i think it is OK to push. | 12:35 |
| Jeffrey4l | fyi, this is another point which we should improve for build.py script. | 12:35 |
| Jeffrey4l | some container will failed for sure. | 12:36 |
| sbezverk | sdake: ping | 12:37 |
| sdake | sbezverk morning | 12:37 |
| *** rwallner has joined #openstack-kolla | 12:37 | |
| sbezverk | sdake: good morning, we need to get 1.6.0 ps merged. | 12:37 |
| sbezverk | I am running recheck and almost all jobs are green | 12:38 |
| sdake | looking | 12:38 |
| sbezverk | even multinodes ones | 12:38 |
| *** ccesario has joined #openstack-kolla | 12:38 | |
| sdake | i saw something about post failures from infra - see subject ;) | 12:38 |
| sdake | sbezverk don't recheck until infra fixes logs.oo plz | 12:39 |
| *** caoyuan has quit IRC | 12:39 | |
| mnaser | ubuntu kolla for 3.0.3 (latest newton release) are being pushed at the moment, should be done shortly | 12:39 |
| sbezverk | sdake: sure, but still you can ack this ps | 12:40 |
| sdake | sbezverk i am | 12:40 |
| sdake | i had a question - not questioning your networking skills - just ramping up mine | 12:40 |
| sdake | https://review.openstack.org/#/c/451391/50/tests/bin/setup_canal.sh | 12:40 |
| sdake | is 130.0 not in a /22? | 12:40 |
| *** dmsimard|afk is now known as dmsimard | 12:41 | |
| sdake | sbezverk looks good once inra disks are fixed and we can get a clean gate run | 12:42 |
| sdake | sbezverk the double kubeadm join needs to be looked at imo | 12:42 |
| sdake | sbezverk i am pretty sure tht awould confuse most software devs (as in they probably dont' test for a double registration and handle it improperly) | 12:43 |
| sdake | may make the mulitnode jobs more reliable or may do nothing | 12:43 |
| sbezverk | sdake: the gate is green, menaing the code is good and does what it is supposed to, we use it in other parts of the code too. I do not see any technical reason to change it.. | 12:44 |
| sdake | sbezverk its redundant | 12:44 |
| sdake | sbezverk it doesn't need to be changed for this patch | 12:45 |
| sdake | i'd +2 the change but all the jobs are in POST_FAILURE | 12:45 |
| sbezverk | sdake: just one | 12:45 |
| mnaser | welp | 12:45 |
| mnaser | i just stopped the ubuntu source builds | 12:45 |
| sbezverk | check current state on zuul | 12:45 |
| mnaser | i mean centos source builds | 12:45 |
| sdake | i count 6 | 12:45 |
| sbezverk | http://status.openstack.org/zuul/ | 12:45 |
| mnaser | tarballs.o.o is affected by the outage :( | 12:46 |
| sbezverk | 451391 in filter | 12:46 |
| sdake | https://review.openstack.org/#/c/451391 | 12:46 |
| sdake | sbezverk lets wiat for logs.oo to be fixed and then get a proper recheck | 12:46 |
| *** schwicht_at_work has quit IRC | 12:47 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Fix Haproxy reconfigure https://review.openstack.org/452166 | 12:48 |
| *** signed8b_ is now known as signed8bit_Zzz | 12:48 | |
| sdake | [05:40:45] <sdake>is 130.0 not in a /22? | 12:48 |
| sdake | sbezverk ^^ | 12:48 |
| sbezverk | will answer in review | 12:49 |
| sdake | just out of curiosity | 12:49 |
| sdake | want ot learn a little more about networking | 12:49 |
| sdake | don't understand how 13.0 worked prior based upon your answer on 132 change | 12:49 |
| sdake | rather 130.0 | 12:50 |
| *** goldyfruit has joined #openstack-kolla | 12:51 | |
| *** dciabrin has joined #openstack-kolla | 12:54 | |
| targon | FYI: I solved my dashboard problem in 4.0.0 by the following: | 12:57 |
| targon | $ sudo docker exec -it horizon bash | 12:57 |
| targon | (horizon)[root@oskollau01 /]# ln -s /var/lib/openstack-dashboard/static /usr/share/openstack-dashboard/static && exit | 12:57 |
| targon | $ sudo docker restart horizon && sudo docker ps | grep horizon | 12:57 |
| targon | This is of cource ugly hack, so still wondering why standard kolla-ansible install failed to insert correct Alias to dashboard.conf | 12:57 |
| *** gkadam has quit IRC | 12:57 | |
| sbezverk | sdake: replied | 12:57 |
| sdake | sbezverk how did 130 work prior? | 12:58 |
| sbezverk | sdake: no idea, why it is that important? we usually do not investigate why somebody made a bug right? Also the explanation I provided is it not sufficient? | 13:00 |
| sdake | sbezverk i was just curious - no idea is a fine answer ;) | 13:00 |
| sdake | sbezverk the reason I want to know is i am developing a deployment doc | 13:01 |
| sdake | and i want to get it right :) | 13:01 |
| sdake | sbezverk i must use the words "no idea" 5 or 6 times in the deployment guide | 13:02 |
| rwellum | sdake: I'm at the business end of the deployment documentation, with some networking questions. Just posted them. | 13:02 |
| rwellum | sdake: 'ni' is more efficient. :p | 13:02 |
| sdake | rwellum infra is busted atm - can't do any updates | 13:02 |
| sdake | i am hesitent to touch anything related to gerrit/etc as to not perturb the recovery | 13:03 |
| sdake | rwellum infra has asked for no submissions/rechecks until logs.oo is fixed | 13:03 |
| *** caoyuan has joined #openstack-kolla | 13:03 | |
| sdake | rwellum see subject set by infra team | 13:04 |
| rwellum | sdake: well they are mainly questions for clarity - they can be updated when things are working. | 13:04 |
| sdake | Jeffrey4l see targon 's workaround for horizon dashboard above | 13:05 |
| rwellum | "set 'ext_interface_name': name of the interface or bridge which will be used by neutron's provider interface" - how do we determine this? | 13:05 |
| sdake | Jeffrey4l i seem to recall we have gone around and around on this | 13:05 |
| sdake | rwellum no idea | 13:05 |
| sdake | rwellum sbezverk knows - he provided the answer to that question :) | 13:06 |
| rwellum | "set 'tunnel_interface': interface name used for connectivity between nodes in kubernetes cluster, in most of cases it matches the name of the kubernetes host management interface" - how do we determine this? | 13:06 |
| *** esharao has joined #openstack-kolla | 13:06 | |
| rwellum | Right. | 13:06 |
| sdake | rwellum on that second one, that should be your "eth0" or your main interface in globals.yaml | 13:07 |
| *** schwicht has joined #openstack-kolla | 13:07 | |
| rwellum | Thanks - I'll suggest that as an update to the doc. | 13:07 |
| sdake | network_interface in globals.yaml | 13:07 |
| *** jtriley has joined #openstack-kolla | 13:08 | |
| sdake | rwellum that is a 90% confidence guess :) | 13:08 |
| sdake | rwellum i think ext_interface_name shoudl be br-ex | 13:08 |
| sdake | rwellum how br-ex is created - unclear | 13:08 |
| rwellum | yeah that one you explain already | 13:08 |
| sdake | rwellum nah, i explained tunnel-interface | 13:09 |
| sdake | "host management interface" = network_interface in globals.yaml | 13:09 |
| rwellum | Right - sorry | 13:09 |
| *** eanylin has quit IRC | 13:10 | |
| sdake | rwellum its ok i was unclear - as its a bit early here and my brain hasn't booted | 13:10 |
| *** gfidente has quit IRC | 13:10 | |
| rwellum | np. So the main unknown is ext_interface_name | 13:11 |
| rwellum | Which is the bridge used by neutron's provider interface | 13:11 |
| rwellum | 50-50 it has to be eth1 | 13:12 |
| sbezverk | rwellum: that is correct | 13:15 |
| *** caoyuan_ has joined #openstack-kolla | 13:15 | |
| rwellum | ty sbezverk | 13:15 |
| sbezverk | rwellum: it is the interface you want to use for your instances external connectivity | 13:15 |
| rwellum | sdake: the two ext_* params are not referenced in your sample cloud.yaml | 13:16 |
| *** lamt has joined #openstack-kolla | 13:17 | |
| *** caoyuan has quit IRC | 13:18 | |
| *** dave-mccowan has joined #openstack-kolla | 13:19 | |
| *** caoyuan_ has quit IRC | 13:19 | |
| *** caoyuan has joined #openstack-kolla | 13:20 | |
| sdake | rwellum didn't say the doc was perfect :) - leave a comment and i'll address as best as poisslbe :) | 13:20 |
| sdake | rwellum i have a cloud.yaml that works but is a big blob of stuff many of which are defaults | 13:20 |
| *** signed8bit_Zzz is now known as signed8b_ | 13:20 | |
| sdake | kfox1111 had asked me to reduce it to the things that are not defaults | 13:20 |
| sdake | kfox1111 we are discovering what those deltas are - I think we have most of them discovered - although if sbezverk could take a look at the review it might be helpful :) | 13:21 |
| rwellum | sdake: I have added all this as comments. | 13:22 |
| sdake | sbezverk if you could review the following for correctness - it would be appreciated: https://review.openstack.org/#/c/447356/ | 13:23 |
| *** Deys has quit IRC | 13:23 | |
| sdake | I think i have everyone's comments collated as the editor of the document | 13:23 |
| sdake | although there have been alot of comments | 13:23 |
| sdake | so I may have got some wrong :) | 13:23 |
| *** caoyuan has quit IRC | 13:24 | |
| *** eanylin has joined #openstack-kolla | 13:27 | |
| rwellum | From your sample cloud.yaml you dropped the openvswitch config section, which is where those two missing params would below. E.g. | 13:27 |
| rwellum | https://www.irccloud.com/pastebin/StZWy54b/ | 13:27 |
| rwellum | I'll add this as a comment. | 13:28 |
| *** caoyuan has joined #openstack-kolla | 13:30 | |
| sdake | sbezverk note i parsed fungi's response to indicate that accessing zuul.openstack.org probably is slowing down the fsck of the disk | 13:34 |
| sdake | zuul.openstack.org reads logs from logs.oo vhost | 13:35 |
| *** Deys has joined #openstack-kolla | 13:35 | |
| *** skramaja_ is now known as skramaja | 13:36 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla master: Use separate sudoers for ironic conductor modprobe https://review.openstack.org/452207 | 13:43 |
| esharao | GM Guys.. | 13:47 |
| esharao | http://paste.openstack.org/show/605069/ | 13:48 |
| esharao | sdake got all the pods running | 13:48 |
| *** gfidente has joined #openstack-kolla | 13:48 | |
| *** gfidente has quit IRC | 13:48 | |
| *** gfidente has joined #openstack-kolla | 13:48 | |
| sdake | esharao morning fine sir | 13:50 |
| sdake | esharao which version of k8s areyou running | 13:50 |
| *** manheim has quit IRC | 13:50 | |
| sdake | esharao nice job btw :) | 13:51 |
| *** manheim has joined #openstack-kolla | 13:51 | |
| esharao | kubeadm version | 13:51 |
| esharao | kubeadm version: version.Info{Major:"1", Minor:"6+", GitVersion:"v1.6.0-alpha.0.2074+a092d8e0f95f52", GitCommit:"a092d8e0f95f5200f7ae2cba45c75ab42da36537", GitTreeState:"clean", BuildDate:"2016-12-13T17:03:18Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64" | 13:51 |
| sdake | esharao your using the RPMs on fedorapeople then? | 13:51 |
| esharao | i followed your guide | 13:51 |
| esharao | yes.. | 13:51 |
| sdake | cool | 13:51 |
| *** Pavo has joined #openstack-kolla | 13:51 | |
| esharao | i had a comment on the change of order for installation of rpms | 13:51 |
| sdake | any deviation from the guide? | 13:51 |
| esharao | yes | 13:52 |
| esharao | i had comments in revision 23 of your guide ;) | 13:52 |
| sdake | esharao - just a heads up - I cannot update the guide until logs.oo is operational again | 13:52 |
| esharao | sure | 13:52 |
| sdake | logs.oo = logs.openstack.org = broken with a corrupted disk - all openstack dev is blocked at present and I don't want to make the problem worse by submitting a review | 13:53 |
| sdake | esharao did you record all deviations from the guide? | 13:53 |
| esharao | yes... | 13:53 |
| *** Serlex has quit IRC | 13:54 | |
| *** manheim has quit IRC | 13:55 | |
| *** manheim has joined #openstack-kolla | 13:55 | |
| *** Pavo has quit IRC | 13:57 | |
| esharao | sdake: should i re-add my comments? | 14:01 |
| *** manheim has quit IRC | 14:01 | |
| *** manheim has joined #openstack-kolla | 14:02 | |
| *** lucasxu has joined #openstack-kolla | 14:03 | |
| *** gfidente has quit IRC | 14:05 | |
| *** shashank_t_ has joined #openstack-kolla | 14:06 | |
| *** manheim has quit IRC | 14:06 | |
| *** Pavo has joined #openstack-kolla | 14:07 | |
| *** shashank_t_ has quit IRC | 14:07 | |
| *** shashank_t_ has joined #openstack-kolla | 14:08 | |
| *** shashank_t_ has quit IRC | 14:09 | |
| *** shashank_t_ has joined #openstack-kolla | 14:09 | |
| *** caoyuan has quit IRC | 14:10 | |
| *** caoyuan has joined #openstack-kolla | 14:11 | |
| sdake | @esharao I guess if I missed them | 14:13 |
| sdake | esharao there are so many comments on that review it is possible I have missed something | 14:14 |
| *** nathharp has joined #openstack-kolla | 14:14 | |
| esharao | ok.. i will re-add them.. its mainly the order of installation of fedorerpms and i think rwellum had similar commnets on git am.. | 14:14 |
| sdake | esharao the latest review I will rev as soon as logs.oo is operational - and publish a new version | 14:14 |
| *** Pavo has quit IRC | 14:14 | |
| esharao | got it | 14:14 |
| sdake | esharao git am is no longer needed | 14:14 |
| sdake | esharao rwellum fixed that problem :) | 14:15 |
| esharao | kool | 14:15 |
| esharao | in general.. can we get some more explanation on the network ip's that folks can use and need to configure.. i feel that is always the confusing part | 14:16 |
| esharao | like what should be the ip in the cloud.yaml and what should be in globals.yaml | 14:16 |
| *** manheim has joined #openstack-kolla | 14:22 | |
| *** Pavo has joined #openstack-kolla | 14:23 | |
| *** lamt has quit IRC | 14:23 | |
| *** gfidente has joined #openstack-kolla | 14:23 | |
| *** gfidente has quit IRC | 14:23 | |
| *** gfidente has joined #openstack-kolla | 14:23 | |
| *** Pavo has quit IRC | 14:24 | |
| *** satyar has joined #openstack-kolla | 14:24 | |
| *** targon has quit IRC | 14:25 | |
| *** zhubingbing has joined #openstack-kolla | 14:26 | |
| *** shashank_t_ has quit IRC | 14:27 | |
| *** shashank_t_ has joined #openstack-kolla | 14:28 | |
| *** new2kolla has joined #openstack-kolla | 14:28 | |
| *** skramaja has quit IRC | 14:29 | |
| hrw | topic is still current? | 14:30 |
| *** zhubingbing__ has joined #openstack-kolla | 14:32 | |
| *** jroll has quit IRC | 14:32 | |
| *** shashank_t_ has quit IRC | 14:32 | |
| *** Pavo has joined #openstack-kolla | 14:33 | |
| nathharp | Hi all - got an 8 node cluster running now, but running into some strange issues when deploying larger numbers of instances (~50-60). Any following instances fail to be created properly, with errors in nova compute: | 14:33 |
| nathharp | BuildAbortException: Build of instance 42651c02-9991-4a01-a5cc-102ea91b4e5e aborted: Failed to allocate the network(s), not rescheduling. | 14:33 |
| nathharp | VirtualInterfaceCreateException: Virtual Interface creation failed | 14:34 |
| *** jroll has joined #openstack-kolla | 14:34 | |
| nathharp | any ideas? | 14:34 |
| *** zhubingbing has quit IRC | 14:35 | |
| egonzalez | nathharp, kolla-ansible or kolla-k8s? | 14:35 |
| nathharp | egonzalez - kolla-ansible | 14:35 |
| egonzalez | nathharp, any error in neutron logs? check quotas, nova had issues with quotas in the first ocata release | 14:36 |
| nathharp | egonzalez - nothing sticks out in the neutron logs, and I’ve set the quotas artificially high | 14:38 |
| egonzalez | nathharp, instances fails in all compute nodes or just in a few? | 14:39 |
| nathharp | egonzalez - with the last test, failed on all but one compute node | 14:39 |
| nathharp | egonzalez - but a previous test it failed on all | 14:40 |
| nathharp | egonzalez - but sometimes it succeeds without problem, and I could run up 120 instances flawlessly | 14:40 |
| egonzalez | nathharp, if you have enabled central logging, try searching for req-<ID> https://github.com/openstack/kolla-ansible/blob/master/doc/central-logging-guide.rst#example-using-kibana-to-diagnose-a-common-failure | 14:42 |
| sdake | hrw indeed | 14:42 |
| egonzalez | maybe that can lead you to where is failing | 14:43 |
| nathharp | ok - I’ll give that a go - I don’t have central logging on at the moment, but I can do | 14:43 |
| nathharp | on that note - is there any documentation for adding nodes or new services etc - ie when ‘deploy’ should be used vs ‘reconfigure’ | 14:44 |
| *** williamwang has joined #openstack-kolla | 14:44 | |
| egonzalez | nathharp, bte, how are you creating that big amount of instances? heat? | 14:44 |
| egonzalez | *btw | 14:44 |
| egonzalez | nathharp, deploy | 14:45 |
| nathharp | just via horizon - creating identical instances booting from image and giving 60 as a quantity | 14:45 |
| egonzalez | nathharp, is failing when creating all together or also individual instances? | 14:46 |
| zhubingbing__ | hey guys | 14:48 |
| zhubingbing__ | egonzalez hi | 14:48 |
| nathharp | egonzalez - very inconsistent, but generally will follow the pattern: create 1 instance (works), create 5 instances (works), create 50 instances (works), delete all instances, create 50 instances (fails), create 1 instance (failes) | 14:48 |
| *** jtriley has quit IRC | 14:52 | |
| *** Pavo has quit IRC | 14:54 | |
| *** manheim has quit IRC | 14:54 | |
| *** manheim has joined #openstack-kolla | 14:55 | |
| *** jtriley has joined #openstack-kolla | 14:55 | |
| *** egonzalez has quit IRC | 14:55 | |
| sdake | fungi gave us green light to submit changes - but to avoid rechecks | 14:56 |
| sdake | hrw sbezverk ^^ | 14:56 |
| *** Deys has quit IRC | 14:56 | |
| sbezverk | sdake: before pushing any other reviews or changes, we need to merge the gate fix.. | 14:58 |
| sdake | sbezverk i am submitting not pushing | 14:58 |
| *** lamt has joined #openstack-kolla | 14:58 | |
| sdake | sbezverk in any regard my change is a docs change | 14:58 |
| sdake | hrw is working on the kolla repo | 14:59 |
| sbezverk | and I did not detect in fungi reply anything about re-checks | 14:59 |
| sdake | that is right in the subject sbezverk | 14:59 |
| zhubingbing__ | hi sdake | 14:59 |
| SamYaple | sbezverk: yea rechecks are fine too | 14:59 |
| sdake | logs.openstack.org has corrupted disks, it's being repaired. Please avoid rechecking until this is fixed | 14:59 |
| zhubingbing__ | sbezverk hi | 14:59 |
| *** manheim has quit IRC | 14:59 | |
| sdake | sup zhubingbing__ | 14:59 |
| sbezverk | zhubingbing__: hey | 14:59 |
| zhubingbing__ | jenkins look bad ；（ | 15:00 |
| *** jmccarthy has left #openstack-kolla | 15:00 | |
| sbezverk | zhubingbing__: yeah infra needs to beefup a bit their servers, I suspect I have more horse power in my home lab ;) | 15:03 |
| zhubingbing__ | roger | 15:03 |
| *** klindgren__ has joined #openstack-kolla | 15:04 | |
| *** klindgren_ has quit IRC | 15:07 | |
| *** schwicht has quit IRC | 15:09 | |
| openstackgerrit | Alicja Kwasniewska proposed openstack/kolla-kubernetes master: Move heat config to kolla-kubernetes https://review.openstack.org/452243 | 15:09 |
| *** eaguilar has joined #openstack-kolla | 15:10 | |
| *** shashank_t_ has joined #openstack-kolla | 15:12 | |
| *** new2openstack has joined #openstack-kolla | 15:12 | |
| *** new2openstack has quit IRC | 15:12 | |
| hrw | sdake: I ended week. only rechecks are waiting | 15:12 |
| *** manheim has joined #openstack-kolla | 15:13 | |
| * hrw off - have a nice weekend | 15:13 | |
| *** sayantan_ has joined #openstack-kolla | 15:24 | |
| *** adrian_otto has joined #openstack-kolla | 15:26 | |
| *** zhubingbing__ has quit IRC | 15:27 | |
| *** jaosorior has quit IRC | 15:28 | |
| *** lucasxu has quit IRC | 15:31 | |
| *** sayantan_ has quit IRC | 15:33 | |
| *** shashank_t_ has quit IRC | 15:34 | |
| *** sayantan_ has joined #openstack-kolla | 15:35 | |
| *** hrw has quit IRC | 15:35 | |
| *** hrw has joined #openstack-kolla | 15:37 | |
| *** pbandark has quit IRC | 15:37 | |
| kfox1111 | morning. | 15:37 |
| kfox1111 | sbezverk: I think the last gate issue is due to placement getting enabled for 3. | 15:39 |
| *** zhubingbing_ has joined #openstack-kolla | 15:41 | |
| sbezverk | kfox1111: good morning, possibly, I cannot access logs any more due to issues with zuul and infra logging | 15:42 |
| kfox1111 | sbezverk: last I got to look at it, it was a rabbit communication issue. | 15:42 |
| kfox1111 | I figured it was a bum run, but hten it occurred to me this morning, | 15:42 |
| kfox1111 | its probably trying to talk to the placement api now that the kolla-ansible patch went in. | 15:43 |
| kfox1111 | just gotta add the placement stuff for 3 as well as 2 I think. | 15:44 |
| sbezverk | kfox1111: and we do not start placement components for this job | 15:44 |
| kfox1111 | right. | 15:44 |
| kfox1111 | sorry. mean,t add 3 as well as 4. | 15:44 |
| sbezverk | kfox1111: as another ps? | 15:44 |
| kfox1111 | probably can tack it on here real quick. as it is a gate fix too. | 15:45 |
| kfox1111 | do you want to do it, or shoudl I? | 15:45 |
| sbezverk | kfox1111: sure, go ahead. I would suggest to do it in a separate ps, this way we can restore gates faster.. | 15:46 |
| *** pcaruana has quit IRC | 15:46 | |
| kfox1111 | gates are broken without the fix though. | 15:46 |
| sbezverk | kfox1111: cause right now all we need is working zuul | 15:46 |
| sbezverk | to restore 90% of gate jobs | 15:47 |
| kfox1111 | depends I guess on how long it will take them to fix zuul. | 15:47 |
| kfox1111 | I think we can get the fix in in the next 10 minutes. | 15:47 |
| sbezverk | kfox1111: ok | 15:47 |
| kfox1111 | (for the 3.x issue) | 15:47 |
| *** deadnull has joined #openstack-kolla | 15:48 | |
| kfox1111 | is it just the placement api? or do we need cell stuff too? | 15:49 |
| deadnull | If I make a change to the globals.yml after a deply, do I need to run deploy, upgrade, or reconfigure? | 15:49 |
| deadnull | deploy* | 15:49 |
| sbezverk | kfox1111: right for 3.X it is just placement | 15:51 |
| kfox1111 | k. | 15:51 |
| kfox1111 | so, I just need to copy the placement stuff from the iscsi job to the ceph one I think. | 15:52 |
| *** shashank_t_ has joined #openstack-kolla | 15:52 | |
| *** lucasxu has joined #openstack-kolla | 15:55 | |
| openstackgerrit | Alicja Kwasniewska proposed openstack/kolla-kubernetes master: Move heat config to kolla-kubernetes https://review.openstack.org/452243 | 15:56 |
| *** caoyuan has quit IRC | 15:57 | |
| *** srwilkers has quit IRC | 15:58 | |
| *** shashank_t_ has quit IRC | 15:58 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Changes for kubernetes 1.6.0 upgrade https://review.openstack.org/451391 | 16:00 |
| kfox1111 | sbezverk: k. this should do it I think. | 16:00 |
| *** fooliouno has joined #openstack-kolla | 16:01 | |
| *** zhubingbing_ has quit IRC | 16:02 | |
| *** zhubingbing_ has joined #openstack-kolla | 16:12 | |
| *** new2kolla has quit IRC | 16:13 | |
| *** shashank_t_ has joined #openstack-kolla | 16:14 | |
| *** manheim has quit IRC | 16:14 | |
| *** nathharp has quit IRC | 16:16 | |
| sdake | kfox1111 i admit i am not a shell wizard | 16:20 |
| sdake | grep 'kubeadm join --token' /tmp/kubeout | awk '{print $4}' > /etc/kubernetes/token.txt | 16:20 |
| sdake | is /tmp/kubeout the output of kubeadm join --token? | 16:20 |
| rwellum | esharao: I have some comments in the latest version of the deploy documentation on the networking in globals and cloud | 16:20 |
| kfox1111 | look for the string 'kubeadm join --token' in the file /tmp/kubeout, and print out the 4th column. | 16:20 |
| sdake | yup i got the awk part :) | 16:21 |
| esharao | great rwellum will take a look | 16:21 |
| sdake | thanks for answering thatquestion | 16:21 |
| sdake | has anyone booted a vm yet? | 16:21 |
| sdake | and had a ping out to the internets? | 16:21 |
| esharao | eanylin did get a vm booted but running into issues with ping | 16:23 |
| sdake | maybe rwellum 's changes fix that | 16:23 |
| sdake | eanylin can you try rwellum's suggestions? | 16:23 |
| eanylin | sdake: In a meeting, ok, will take a look | 16:24 |
| sdake | eanylin thanks - hopefully i'll have the review integrated prior to your meeting ocnclusion | 16:24 |
| eanylin | actually I wasnt able to ping the VM from the host machine | 16:24 |
| eanylin | or SSH to it | 16:24 |
| sdake | eanylin did you add a floating ip? | 16:24 |
| eanylin | sdake: Yes, I did | 16:25 |
| inc0 | how is my favorite community doing today? | 16:25 |
| eanylin | sdake: I got busy with work these 2 days, didnt really look into the ping/SSH issue | 16:25 |
| sdake | inc0 TGIF | 16:25 |
| sdake | eanylin thats ok eanylin your pioneering all the deployment guide work ;) | 16:26 |
| sdake | I have trouble keeping up with all the comments as an editor :) | 16:26 |
| sdake | much less personally testing them | 16:26 |
| sdake | which I hane't done since wednesday | 16:26 |
| sdake | inc0 after that goland course do you feel smarter? | 16:27 |
| eanylin | sdake: Glad I helped :) | 16:27 |
| fooliouno | I entered a comment on the deployment guide and it stays in a "draft" state. Is that how its supposed to be? | 16:27 |
| inc0 | negative, stupider is more accurate | 16:27 |
| inc0 | but I learned a lot so that's good | 16:27 |
| sdake | inc0 yup i thought you'd say soething like that :) | 16:28 |
| sdake | eanylin any help you can provide in the future is valued :) | 16:28 |
| sdake | inc0 as soon as this kubenertes 1.6.0 change is passing its giongto need an ack from you and me | 16:28 |
| sbezverk | kfox1111: I do not recall if I had to use source, to get placement working.. | 16:29 |
| *** lucasxu has quit IRC | 16:29 | |
| inc0 | sdake: link plz | 16:29 |
| eanylin | sdake: Yeah, I will continue to help | 16:29 |
| *** srwilkers has joined #openstack-kolla | 16:30 | |
| kfox1111 | sbezverk: oh. you think its broken in 3? | 16:31 |
| sdake | https://review.openstack.org/#/c/451391/ | 16:31 |
| *** satyar has quit IRC | 16:32 | |
| sbezverk | kfox1111: there were a couple of breakage which forced me to use source, I just do not recall if placement was one of them | 16:32 |
| sdake | inc0 add yourself as a review and keep an eye on that - we need to merge it as soon as infra is runningagain | 16:32 |
| *** Pavo has joined #openstack-kolla | 16:32 | |
| kfox1111 | sbezverk: ah. ook. well, I guess we'll have to see. | 16:32 |
| kfox1111 | worst case, we can just cut out the placement stuff from the nova config. | 16:32 |
| sdake | kfox1111 we need the placement api for 4.0.0 images | 16:33 |
| sbezverk | kfox1111: or switched to source ;) | 16:33 |
| kfox1111 | sdake: yeah. it would be conditional. | 16:33 |
| kfox1111 | sbezverk: -1 | 16:33 |
| sdake | kfox1111 wfm :) | 16:33 |
| sdake | ya we want to gate on binary and source | 16:33 |
| kfox1111 | sdake: I got gate jobs for 4.x into zuul, so wenever the gate's stable, we can start making 4.x jobs. | 16:34 |
| sdake | kfox1111 nice | 16:34 |
| sdake | kfox1111 experimental or check jobs? | 16:35 |
| *** rwallner has quit IRC | 16:35 | |
| sdake | ok - sine we pretty much treat the gate as voting | 16:35 |
| sdake | why nto make it voting for all current check jobs? | 16:36 |
| kfox1111 | sdake: experimental until they are written/stable. | 16:36 |
| sdake | we had this discussion briefly before | 16:36 |
| *** Pavo has quit IRC | 16:36 | |
| *** rwallner has joined #openstack-kolla | 16:36 | |
| sdake | and i heard conflicting reports on what the core reviewers wanted | 16:36 |
| kfox1111 | sdake: you said you would submit that patch I think. | 16:36 |
| sdake | kfox1111 sbezverk said no to 2.0.0 images | 16:36 |
| sdake | and i htink binary as well although i dont recall | 16:36 |
| sdake | i will submit the patch after i get clear consensus on what should be voting | 16:37 |
| kfox1111 | k | 16:37 |
| kfox1111 | mitaka's our most well tested stuff. and a lot of ops are still using it. | 16:37 |
| kfox1111 | I think we should be user servey based until we hit 1.x. | 16:37 |
| sdake | user survey based - expand? | 16:38 |
| fooliouno | kfox1111: Did you (or anyone you know on this channel) try bringing up odl on kolla-kubernetes | 16:38 |
| kfox1111 | our focus is on making it easy for operators to adopt our stuff, right? | 16:38 |
| kfox1111 | fooliouno: not yet. | 16:38 |
| fooliouno | ok .. was going to look into that. | 16:39 |
| kfox1111 | sdake: as such, we should focus on what operator data is telling us from the user servey. | 16:39 |
| sbezverk | kfox1111: with 2.X images we have very limitted feature support and only ceph backend, so basically we re testing old stuff which is not even supported any more | 16:39 |
| sdake | kfox1111 are you suggesting people will migrate from bare metal to kubernetes? | 16:39 |
| sbezverk | kfox1111: I get your point of upgrade etc, but it should not be the main testing image | 16:40 |
| sbezverk | maybe as experimental job for upgrade, yes, but main gate must bed 4.X | 16:40 |
| *** mgoddard has quit IRC | 16:40 | |
| kfox1111 | sdake: yes. I know of at least 3 clouds planning on it. | 16:41 |
| kfox1111 | sdake: fuel left a lot of clouds behind too. | 16:41 |
| sdake | kfox1111 is that a thing that can be done? | 16:41 |
| sdake | newton is currently a branch in kolla repo | 16:42 |
| kfox1111 | sdake: non trivial for sure, but I'm going to do it, as I think one of my coworkers is looking at it. | 16:42 |
| sdake | (kolla=containers) | 16:42 |
| kfox1111 | sbezverk: I agree, we shoudl move stuff to 4. but that doesn't mean we shoulnt vote on 2. | 16:42 |
| kfox1111 | operators hardly ever run N. thats what the user servey says. | 16:43 |
| kfox1111 | they ofen run N-1, N-2, | 16:43 |
| sdake | kfox1111 once newton is unmaintained (things become EOLed in openstack) I have a tough time rationalizing keeping that in the gate | 16:43 |
| kfox1111 | and if somethings just released, even N-3 for abit. | 16:43 |
| sbezverk | kfox1111: if having 2 and 4 do not extend gate job running time, then it should be ok | 16:43 |
| sdake | sbezverk they run almost in parallel | 16:44 |
| kfox1111 | sbezverk: yeah. should be parallel deploys. | 16:44 |
| sbezverk | because the most of breakage will be on more recent versions | 16:44 |
| kfox1111 | yeah. | 16:44 |
| sdake | i am more concerned about not being able to ship stable newton kolla container security updates | 16:44 |
| kfox1111 | sdake: I think thats the operators concern. | 16:44 |
| kfox1111 | operators do choose to run less then secure systems at times, due to nessesity. | 16:45 |
| kfox1111 | we have mechanisms in place to mitigate the issues. | 16:45 |
| kfox1111 | kolla though should try and provide as much as possible, up to date containers though. | 16:46 |
| kfox1111 | brb | 16:46 |
| sdake | ok well lets separate the discussion of gating from what is shippable as a security update | 16:47 |
| sdake | we dont know precisely when newton eols | 16:47 |
| sdake | when inc0 lets us know, we can have a wider discussion on that topic | 16:47 |
| sdake | for the moment, are there negatives to making all the current check gates voting? | 16:47 |
| * inc0 reading up | 16:48 | |
| inc0 | I think that will also resonate with "better tagging" discussion | 16:49 |
| inc0 | and automatic infra push jobs | 16:49 |
| inc0 | and frankly? I think this is spec-worthy | 16:50 |
| sdake | think which is spec worthy | 16:50 |
| sdake | all we are talking about is making votes gating | 16:50 |
| inc0 | ok, I misunderstood then | 16:50 |
| inc0 | votes gating or voting gates?;) | 16:51 |
| * sdake run ragged by his children | 16:52 | |
| sdake | inc0 voting gates | 16:52 |
| inc0 | in kk8s? | 16:52 |
| sdake | yup | 16:52 |
| inc0 | I'd say we need to move kk8s to master images then | 16:53 |
| inc0 | and make sure they're stable | 16:53 |
| sdake | as soon as we have dockerhub images that are master, we can start the process of making master checks | 16:54 |
| inc0 | or use tarballs.o.o | 16:54 |
| *** lucasxu has joined #openstack-kolla | 16:54 | |
| sdake | tarballs.oo is not viable for master images | 16:55 |
| sdake | (without a whole lot of work) | 16:55 |
| sdake | anyway lets stay focused on the gates we have | 16:55 |
| sdake | gating or not gating that is the question :) | 16:55 |
| inc0 | to me master images are must have for voting gates | 16:56 |
| inc0 | because if something goes wrong, we need to be able to fix stuff easily | 16:56 |
| inc0 | that's 1 | 16:56 |
| inc0 | 2 not whole lot of work | 16:57 |
| inc0 | literally 3 commands + k8s config | 16:57 |
| sdake | ok well in the meantime while that master gating gets sorted out (by who?) | 16:57 |
| inc0 | download tarball -> untar somewhere -> docker run registry -v /var/lib/regisstry:somewhere | 16:58 |
| sdake | the question is which of the 2.0.0/3.0.0/4.0.0 gates should be voting | 16:58 |
| inc0 | none, again either we don't mechanism to ship upgrades and our gates are worthless as operators won't use what's in dockerhub | 16:58 |
| inc0 | (and I don't want to push change to kolla-k8s gates every time we release z) | 16:59 |
| sdake | one change every 45 days is not that significant | 16:59 |
| inc0 | or we gate on stale images and gates are worthless because real world scenerio will not use stale images | 16:59 |
| inc0 | it's more than doing it properly in the first place | 16:59 |
| sdake | inc0 i think we need to wait for kfox1111 to return | 17:00 |
| sdake | where we are currently is working on 4.0.0 gating | 17:00 |
| sdake | which is a necessary thing | 17:01 |
| inc0 | great, but not voting | 17:01 |
| sdake | why not voting? | 17:01 |
| inc0 | and temporary | 17:01 |
| *** signed8b_ is now known as signed8bit_Zzz | 17:01 | |
| sdake | they are treated as voting | 17:01 |
| inc0 | because if something goes wrong we have a lot of work to get gates back into order | 17:01 |
| inc0 | treated is fine, blocked on infra is not | 17:01 |
| sdake | kolla-kubernetes has a different set of gating problems then either kolla or kollaansible | 17:01 |
| inc0 | I'm well aware | 17:02 |
| sdake | we have blocked on infra for the last 4 days waiting for kubernete 1.6 to get merged | 17:02 |
| sdake | maybe 3 days | 17:02 |
| inc0 | I'm also well aware how many times we had to hacky-upload something to dockerhub because kk8s needed it | 17:02 |
| inc0 | that's outside of our control | 17:02 |
| inc0 | images aren't | 17:02 |
| inc0 | I think we need to get rid of this mix and match of images in kk8s asap | 17:03 |
| sbezverk | I think voting on binary is wrong, because in this case we rely our decision on vendor packaging | 17:03 |
| sbezverk | which we have no control over at all | 17:03 |
| inc0 | that's why we don't have voting binary in kolla itself | 17:03 |
| sbezverk | with source, at least we have access to openstack developers for this specific component | 17:04 |
| sdake | inc0 that decision (no voting on binary in kolla) makes sense but is orthognoal to this discussion | 17:04 |
| sbezverk | inc0: agree, I think it makes sense | 17:04 |
| inc0 | commenting on sbezverk's note | 17:04 |
| sdake | which hacky workarounds did we upload to dockerhub to fix the check gates? | 17:05 |
| inc0 | so yes, I'm all for voting gates as long as they follow same methodology as all other deliverables | 17:05 |
| sdake | the only example I can think of is to implement *experimental* gates | 17:05 |
| *** srwilkers has quit IRC | 17:05 | |
| inc0 | from zen of python Special cases aren't special enough to break the rules. | 17:05 |
| *** athomas has quit IRC | 17:06 | |
| rwellum | Has anyone had this issue before? | 17:06 |
| rwellum | https://www.irccloud.com/pastebin/LFwtnU85/ | 17:07 |
| rwellum | I do have that namespace created | 17:07 |
| rwellum | https://www.irccloud.com/pastebin/HWRAzoGW/ | 17:07 |
| sdake | rwellum are you using kubernets 1.6.0? | 17:07 |
| rwellum | y | 17:07 |
| sdake | 1.5.4 works for others | 17:08 |
| sdake | are you really keen to get 1.6.0 kubernetes running? | 17:08 |
| rwellum | I got further with 1.6.0 than 1.5.4 | 17:09 |
| sdake | sbezverk hvae you seen rwellum 's bug during the kubernetes 1.6.0 changes? | 17:10 |
| sdake | rwellum note the guide doesn' thave the correct information to deploy kolla-kubernetes on 1.6.0 | 17:10 |
| sdake | version madness ftw :) | 17:11 |
| rwellum | Yeah I have a mashup of your doc and sbezverk fixes | 17:11 |
| rwellum | It's easy to try 1.5.4 again but I was making progress | 17:12 |
| *** sambetts is now known as sambetts|afk | 17:12 | |
| rwellum | sbezverk: has stepped out for a couple hrs fyi sdake | 17:12 |
| sdake | rwellum i'd suggest we hammer out 1.5.4 first, then we can do 1.6.0 next :) | 17:15 |
| *** signed8bit_Zzz is now known as signed8b_ | 17:16 | |
| rwellum | Ack sdake - I thought with sbezverk work on the gate 1.6 was ready. esharao what have you been trying? You are further than me... | 17:16 |
| esharao | rwellum, i gave up on 1.6 got now.. went back to older one.. will try 1.6 over hte weekend | 17:17 |
| *** shardy has quit IRC | 17:17 | |
| *** mliima__ has joined #openstack-kolla | 17:17 | |
| rwellum | fair enough esharao | 17:18 |
| *** mliima_ has quit IRC | 17:19 | |
| *** mgoddard has joined #openstack-kolla | 17:20 | |
| sdake | rwellum we dont know if its ready or not as infra is busted | 17:21 |
| openstackgerrit | Marcus Williams proposed openstack/kolla-ansible master: Add OpenDaylight role https://review.openstack.org/416367 | 17:22 |
| kolla-slack | <jascott1> o/ | 17:28 |
| *** shashank_t_ has quit IRC | 17:34 | |
| *** shashank_t_ has joined #openstack-kolla | 17:34 | |
| *** sayantan_ has quit IRC | 17:39 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla-kubernetes master: Updated from global requirements https://review.openstack.org/451018 | 17:41 |
| *** new2kolla has joined #openstack-kolla | 17:43 | |
| openstackgerrit | Marcus Williams proposed openstack/kolla-ansible master: Split Openvswitch into own role https://review.openstack.org/416793 | 17:44 |
| *** eaguilar has quit IRC | 17:45 | |
| *** MasterOfBugs has joined #openstack-kolla | 17:48 | |
| openstackgerrit | Marcus Williams proposed openstack/kolla-ansible master: Add OpenDaylight role https://review.openstack.org/416367 | 17:48 |
| *** srwilkers has joined #openstack-kolla | 17:53 | |
| *** Pavo has joined #openstack-kolla | 17:54 | |
| *** Pavo has quit IRC | 17:55 | |
| *** zhubingbing_ has quit IRC | 17:57 | |
| *** unicell has joined #openstack-kolla | 17:58 | |
| *** schwicht has joined #openstack-kolla | 18:00 | |
| *** new2kolla_ has joined #openstack-kolla | 18:00 | |
| *** new2kolla has quit IRC | 18:01 | |
| *** bswartz has joined #openstack-kolla | 18:03 | |
| kfox1111 | back | 18:09 |
| *** lucasxu has quit IRC | 18:12 | |
| *** mgoddard has quit IRC | 18:13 | |
| mnaser | FYI 3.0.3 images for ubuntu binary and source have been published | 18:15 |
| mnaser | centos is still waiting becaus i cant build source images | 18:15 |
| mnaser | tarballs.o.o is having issues because of the logs issues | 18:16 |
| kfox1111 | inc0: I think we need to focus on getting image building in gate at the summit. | 18:16 |
| *** Pavo has joined #openstack-kolla | 18:19 | |
| inc0 | kfox1111: please dont... | 18:19 |
| inc0 | you want to build images in deploy gates? that'll raise complexity of gates a lot | 18:19 |
| kfox1111 | inc0: no. release gates. | 18:19 |
| inc0 | what's a release gate? what do you mean? | 18:20 |
| kfox1111 | like, we build, test, and release to docker hub in a job. | 18:20 |
| *** Pavo has quit IRC | 18:20 | |
| mnaser | kfox1111 the work that i hope to finish regarding getting a publisher should help with this | 18:20 |
| mnaser | but thats a lot of traffic to do for each commit | 18:20 |
| kfox1111 | mnaser: yeah, but not per commimt. per release. | 18:21 |
| mnaser | oh yeah | 18:21 |
| mnaser | right now its done manually, but i started some work to give us the ability to do it per release | 18:21 |
| kfox1111 | we tag 3.0.3, and it builds all the thigns. :) | 18:21 |
| mnaser | https://review.openstack.org/#/c/447524/ | 18:21 |
| kfox1111 | right. :) | 18:21 |
| mnaser | im letting the discussion brew | 18:21 |
| mnaser | and send a revision | 18:21 |
| kfox1111 | yeah. that review. :) | 18:21 |
| kfox1111 | we should work on it at the summit. | 18:21 |
| mnaser | im hoping people come to terms with the "but zuulv3 will solve this" | 18:21 |
| mnaser | (i dont want to be the person who asks "when will zuulv3 be out to know if we can justify this") | 18:21 |
| mnaser | :p | 18:21 |
| kfox1111 | corner some infra folks. :) | 18:21 |
| kfox1111 | right. | 18:22 |
| kfox1111 | zuul3 has been promiced for a while. | 18:22 |
| kfox1111 | though I think they have a version in place now. just not everythign migrated. | 18:22 |
| mnaser | they do | 18:22 |
| kfox1111 | so we might be able to start a new job type over there. | 18:22 |
| mnaser | i believe zuul self tests with zuulv3 | 18:22 |
| kfox1111 | we could be earlier adopters potentially. | 18:22 |
| kfox1111 | cool. | 18:22 |
| mnaser | yeah | 18:24 |
| mnaser | i dont mind that, as an experimental thing | 18:24 |
| mnaser | we could setup a separate testing namespace for kolla for that job | 18:24 |
| kfox1111 | :) | 18:25 |
| *** jascott1_ has joined #openstack-kolla | 18:26 | |
| rwellum | sdake: 1.5.x - mixed bag, quite a few of the helm charts succeeded, quite a few are not up yet, about 30m in. https://www.irccloud.com/pastebin/ekEIFgUH/ | 18:29 |
| kfox1111 | + | 18:34 |
| sbezverk | so what have you decided ;) ? | 18:35 |
| *** sayantan_ has joined #openstack-kolla | 18:35 | |
| *** manheim has joined #openstack-kolla | 18:37 | |
| kfox1111 | sbezverk: who was that directed at? | 18:37 |
| sbezverk | kfox1111: to the universal wisdon ;) | 18:37 |
| sbezverk | wisdom sorry | 18:37 |
| kfox1111 | ah. :) | 18:38 |
| sbezverk | kfox1111: I left when we were talking about binary voting, which I personally hate | 18:38 |
| kfox1111 | oh. my random keystroke? :) | 18:38 |
| kfox1111 | sbezverk: yeah. I feel the same way about source. | 18:38 |
| kfox1111 | I feel more comfortable with redhat/ubuntu tested stuff, then upstream only tested stuff. | 18:39 |
| sbezverk | kfox1111: cool, then here is the scenario centos binary fails ubuntu binary broken, do we fail the change? | 18:39 |
| kfox1111 | absolutely. :) | 18:39 |
| kfox1111 | because our users will absolutely be trying it themselves, and we don't want to break them. | 18:39 |
| sbezverk | but why? in thi case we can get stuck for long long time | 18:39 |
| kfox1111 | then we better work with upstream for the binaries to fix it. | 18:40 |
| kfox1111 | because thats what our users will need. | 18:40 |
| sbezverk | until centos fixes | 18:40 |
| sbezverk | its packaging | 18:40 |
| kfox1111 | thats one reason we should tag our releases and keep them around. | 18:40 |
| kfox1111 | if 3.0.3-1 works, and 3.0.3-2 breaks due to upstream package, | 18:40 |
| sbezverk | yeah, but it is not | 18:40 |
| kfox1111 | we can keep using 3.0.3-1 until upstream fixes, and we cut a final, workable 3.0.3-2. | 18:41 |
| *** manheim has quit IRC | 18:41 | |
| kfox1111 | I think most of the problems we've hit with binaries were lack of release procedure around building the containers. | 18:41 |
| kfox1111 | that should be solvable. | 18:42 |
| kfox1111 | allowing us to break against binaries though when users rely on binaries isn't a good plan. | 18:42 |
| sbezverk | kfox1111: well, our primary goal is develop and delay this process based on unrelated to our works things does not sound right to me. I would suggest to put it on vote.. | 18:47 |
| kfox1111 | ah. dev vs ops battle. :) | 18:48 |
| kfox1111 | our primary goal is to produce something operators would like to use. ;) | 18:48 |
| kfox1111 | further development is an important goal. | 18:48 |
| sbezverk | kfox1111: we should test binaries, but it should not be decisive especially if the same change fopr source passes | 18:48 |
| kfox1111 | but not alienating the users is also another important goal. | 18:49 |
| kfox1111 | non voting gates tend to regress into alwasy broken. | 18:49 |
| kfox1111 | and non tested=broken. | 18:49 |
| kfox1111 | not a good place to be. | 18:49 |
| kfox1111 | caching known good binaries should mostly eliviate the issue. | 18:50 |
| kfox1111 | as will, post 1.x, doing older versions in branches. | 18:50 |
| kfox1111 | until 1.x though, I think its better to keep all the versions working/advancing together. | 18:50 |
| kfox1111 | we need build pipelines too before we split them off. | 18:51 |
| sbezverk | kfox1111: I am ok with building more jobs for better testing since number does not extend job's running time, but I do not want to end up in situation, that the change I need to merge, does not getting merge for the reason that has absolutely nada with me :) | 18:53 |
| kfox1111 | sbezverk: I understand that. very sympothetic to it. | 18:53 |
| kfox1111 | but also sympotentic to operators not wanting broken stuff pushed to them all the time. :/ | 18:54 |
| kfox1111 | like the 1.6.0 feasco. | 18:54 |
| kfox1111 | what we need to do is figure out a nice middle ground where the chances of binaries breaking things is minimal. | 18:55 |
| kfox1111 | I think most issues we have hit were things not in the source build? | 18:55 |
| kfox1111 | sorry... let me rephrase that... | 18:55 |
| kfox1111 | issues were with specific builds of containers, not whether the source of openstack came from rpm or pip. | 18:55 |
| kfox1111 | so we need more control over our own building of containers process. | 18:56 |
| kfox1111 | which is what I've been pushing for. | 18:56 |
| kfox1111 | right now its very very manual. | 18:57 |
| kfox1111 | and very uncommon. | 18:58 |
| sbezverk | kfox1111: past showed that most of issue with container were wrong versions of software packages | 18:58 |
| kfox1111 | we need frequent container building. | 18:58 |
| kfox1111 | sbezverk: yeah. but again, stuff like libvirt. ddidn't matter if its centos-source or centos-binary? | 18:58 |
| kfox1111 | the real issue is we're not frequiently updating the containers I think, to get binary bug fixes. | 18:58 |
| sbezverk | if repo has outdated version of rabbitmq and it breaks everything else, good luck, it could be days before it gets changed | 18:59 |
| kfox1111 | why would it spontaniously break? | 18:59 |
| kfox1111 | if the outdated container use to work, why would it suddenly stop working? | 18:59 |
| kfox1111 | I'm maybe not following something. | 19:00 |
| sbezverk | kfox1111: because somebody else, let's say neuton pushed new rpm which is using 'new' rabbitmq feature | 19:00 |
| *** lucasxu has joined #openstack-kolla | 19:00 | |
| kfox1111 | binary woudlnt do that. in fact, often source does. | 19:00 |
| kfox1111 | cause binaries test the whole set together. | 19:00 |
| kfox1111 | which is why I like them. | 19:00 |
| sbezverk | well I have first hand experience | 19:01 |
| kfox1111 | that specific senerio? | 19:01 |
| sbezverk | binary centos ironic 3 worked | 19:01 |
| sbezverk | ubuntu binary failed | 19:01 |
| sdake | rwellum im not sure - i need to head out for a bit - I'll work on getting that patch tested in my environment when i submit the new rev | 19:01 |
| sbezverk | but both sources worked like a charm | 19:01 |
| kfox1111 | I'd attribute that to an outdated 3 build, not binaries. | 19:01 |
| kfox1111 | it wasn't source that fixed it, | 19:02 |
| *** lucasxu has quit IRC | 19:02 | |
| kfox1111 | it was when it was built. | 19:02 |
| sdake | kfox1111 ubuntu binary is always a pile of garb | 19:02 |
| sdake | kfox1111 has been since day one | 19:02 |
| kfox1111 | disagree. its worked pretty reliably in the gate. | 19:02 |
| sdake | the core services work fine | 19:02 |
| sdake | anything outside what ubuntu considers core (such as ironic) does not work well | 19:03 |
| sdake | whereas source alwaysworks well with kolla | 19:03 |
| kfox1111 | that may be. no experience with it. | 19:03 |
| kfox1111 | if that proves to be the case with ubuntu in kolla-kubernetes, I would vote to not gate the binaries for that stuff. | 19:03 |
| sdake | anyway you guys shoudl have a death match battle with inc0 and make a decision | 19:03 |
| sdake | inc0 doens't want any voting gating at all | 19:03 |
| sdake | so YMMV :) | 19:03 |
| kfox1111 | heh. | 19:03 |
| *** imcsk8 has quit IRC | 19:03 | |
| kfox1111 | if core ubuntu binaries are stable though, we should gate them. | 19:04 |
| *** Pavo has joined #openstack-kolla | 19:04 | |
| *** Pavo has quit IRC | 19:04 | |
| kfox1111 | redhats binaries are stable. so lets gate them too. | 19:04 |
| kfox1111 | the main issue is really stale containers. | 19:04 |
| kfox1111 | lets fix that. | 19:04 |
| inc0 | it's not that I don't want voting gates | 19:04 |
| inc0 | I want voting based of master and stable branches | 19:05 |
| inc0 | not dockerhub pulled mix | 19:05 |
| kfox1111 | inc0: I agree with that, long term. | 19:05 |
| inc0 | so yes, tarballs.o.o is an answer to that | 19:05 |
| inc0 | and instead of spending time hacking some uploading script | 19:05 |
| inc0 | let's just finish tarballs-pulled gates | 19:05 |
| *** jascott1_ has quit IRC | 19:06 | |
| kfox1111 | inc0: yeah. was almost done with that. | 19:06 |
| inc0 | exactl7 | 19:06 |
| inc0 | it's not a big deal to get it working | 19:06 |
| kfox1111 | was hoping to finish that, but 1.6 landed. :/ | 19:06 |
| inc0 | ofc | 19:06 |
| kfox1111 | and that fixes the thing I've been squawking about. we need a way to refresh containers. | 19:06 |
| kfox1111 | that gives us a mechanism for it. | 19:06 |
| inc0 | exactly my point | 19:07 |
| kfox1111 | tarballs.o.o has the most up to date, built and tested containers. | 19:07 |
| inc0 | and local to nodepools so we're lowering bandwidth requirement of gates | 19:07 |
| inc0 | and issues due to it | 19:07 |
| kfox1111 | that woudl be nice. | 19:07 |
| inc0 | that's how it works | 19:07 |
| kfox1111 | to lower bandwidth, we need container fingerprinting too. | 19:08 |
| inc0 | well in gates we need to pull whole stack anyway | 19:08 |
| kfox1111 | yeah, but to upload updated containers, while minimizing bandwidth, | 19:08 |
| kfox1111 | we need fingerprints. | 19:08 |
| inc0 | let's not concern ourselves with that yet | 19:08 |
| inc0 | we're uploading whole stack | 19:08 |
| inc0 | all the time | 19:08 |
| inc0 | not a huge deal | 19:09 |
| kfox1111 | yeah, but sooner rather hten later. | 19:09 |
| inc0 | I want dockerhup autopublish too | 19:09 |
| kfox1111 | +1 | 19:09 |
| kfox1111 | we should work on that at the summit. | 19:09 |
| inc0 | just frankly? lower priority than stable gates for kolla k8s | 19:09 |
| kfox1111 | +1 | 19:09 |
| kfox1111 | the only reason we should start now on it thoguh, | 19:09 |
| kfox1111 | is its wall clock expensive. | 19:10 |
| inc0 | http://tarballs.openstack.org/kolla/images/ <- we have master and ocata | 19:10 |
| kfox1111 | might take months of simmering to finish. | 19:10 |
| openstackgerrit | Marcus Williams proposed openstack/kolla-ansible master: Add OpenDaylight role https://review.openstack.org/416367 | 19:10 |
| inc0 | well good alternative (or even better) would be registry.openstack.org | 19:10 |
| kfox1111 | inc0: http://tarballs.openstack.org/kolla-kubernetes/gate/containers/ <--- we have 2 and 3 also. ;) | 19:10 |
| kfox1111 | tested by kolla-kubernetes. | 19:11 |
| kfox1111 | along with a tested set of k8s containers. :) | 19:11 |
| inc0 | can we narrow it down to single dir please | 19:11 |
| inc0 | also I'd stick to ocata | 19:11 |
| inc0 | with kolla-k8s | 19:11 |
| kfox1111 | inc0: once cross repo gating is done, that may be possible. | 19:12 |
| inc0 | with all the nova-placement stuff | 19:12 |
| *** imcsk8 has joined #openstack-kolla | 19:12 | |
| kfox1111 | kolla's stuff isn't kolla-kubernetes tested. | 19:12 |
| kfox1111 | the kolla-kubernetes/gate containers are fully tested. | 19:12 |
| inc0 | yeah, that's importnant | 19:12 |
| kfox1111 | -1 to just ocata. | 19:12 |
| inc0 | but not kolla-tested ;) | 19:12 |
| kfox1111 | inc, kolla tested too. | 19:12 |
| kfox1111 | they come from kolla. | 19:12 |
| inc0 | as in uploaded to dockerhub | 19:12 |
| inc0 | but potentially stale | 19:13 |
| kfox1111 | right. they are sourced from dockerhub. | 19:13 |
| inc0 | that's my point | 19:13 |
| kfox1111 | right. | 19:13 |
| inc0 | to me stale = soft-broken | 19:13 |
| kfox1111 | we need to smooth out the workflow quite a bit. | 19:13 |
| kfox1111 | yeah, but less then hard broken which often comes out. :/ | 19:13 |
| inc0 | I'd start by adding non-voting gates on master and ocata tarballs | 19:13 |
| inc0 | fix if need fixing | 19:13 |
| inc0 | then add cross-repo to kolla and make gates voting | 19:13 |
| kfox1111 | inc0: thats in progress. | 19:13 |
| inc0 | this order of business | 19:14 |
| kfox1111 | yup. | 19:14 |
| inc0 | as for gates on newton we might just manually create+upload tarball for images | 19:14 |
| inc0 | we're not backporting auto-upload code | 19:14 |
| kfox1111 | so.... | 19:15 |
| inc0 | so...can we restart https://review.openstack.org/#/c/418550/ | 19:15 |
| inc0 | ? | 19:15 |
| kfox1111 | I'm mostly interested in older versions, as the user survey says most people are runningn older versions. | 19:15 |
| kfox1111 | they do that, because its a pain to upgrade. | 19:15 |
| inc0 | stable branches are most important | 19:15 |
| kfox1111 | support, although with stale containers, | 19:15 |
| kfox1111 | can be used as a stepping stone to migrate forward. | 19:15 |
| inc0 | but changing underlay to k8s will be harder than upgrade.. | 19:15 |
| kfox1111 | to a supported container set. | 19:16 |
| inc0 | or rather...equally as hard for worst case | 19:16 |
| kfox1111 | yes. but a sideways upgrade from non k8s to k8s is way easier then an upgrade from not k8s to k8s, + a n+1 upgrade at the same time. | 19:16 |
| Mech422 | inc0: got a minute to talk about the etcd/confd stuff ? | 19:17 |
| kfox1111 | so continued support of mitaka for example, just with stale containers is a stepping stone to onboarding brownfield users. | 19:17 |
| inc0 | a minute, need to drive my wife to airport soon | 19:17 |
| Mech422 | inc0: Ohh! baching it! Weekend Blowout at inc0's house!! | 19:17 |
| inc0 | kfox1111: yeah...that being said, I'd rather onboard brownfield when we're better tested on greenfield | 19:17 |
| * Mech422 lives the brownfield life.... :-( | 19:18 | |
| inc0 | brownfield will be good use case for kolla-ansible tbh | 19:18 |
| kfox1111 | I know of 3 brownfield clouds thinking about kolla-kubernetes. | 19:18 |
| inc0 | kfox1111: while you can handle migrating to kolla-k8s because you know this stuff in and out | 19:18 |
| inc0 | well I'm not saying not a priority | 19:19 |
| kfox1111 | yup. but operators are a resilient bunch. | 19:19 |
| inc0 | I'm saying I'm afraid to push it | 19:19 |
| inc0 | brownfield is much harder than greenfield | 19:19 |
| kfox1111 | agreed. | 19:19 |
| inc0 | and we didn't solve greenfield yet | 19:19 |
| kfox1111 | but if we can't handle brownfield, we won't properlty handle greenfield either. | 19:19 |
| inc0 | I'd be more comfortable to push kolla-ansible or kolla-your-config-mgmt | 19:19 |
| kfox1111 | no other distro that focused on just greenfield has. :/ | 19:19 |
| Mech422 | inc0, kfox1111 so far, my experience with brownfield has been the politics are harder then the tech... | 19:20 |
| kfox1111 | Mech422: yeah. agreed. | 19:20 |
| inc0 | hehe ain't that true | 19:20 |
| kfox1111 | brownfield is easier/harder in someways. | 19:20 |
| kfox1111 | depends on the skill of the op. | 19:20 |
| inc0 | we have brownfield notes for kolla-ansible I'm quite fond of | 19:20 |
| inc0 | which can be reused for most part | 19:20 |
| inc0 | but will require lots and lots of hacking on k8s | 19:21 |
| kfox1111 | sure. | 19:21 |
| kfox1111 | my focus on brownfield is for ops that built their own snowflakes at this point. | 19:21 |
| Mech422 | kfox1111: special snowflak Mech reporting for duty! | 19:22 |
| kfox1111 | if they know enough to do that, then they will be in pretty good shape. | 19:22 |
| kfox1111 | :) | 19:22 |
| inc0 | and you are assurance that we'll not break this use case:P | 19:22 |
| kfox1111 | they don't nneed complex greenfield level orchestration tools that are buggy/incomplete. | 19:22 |
| *** jascott1_ has joined #openstack-kolla | 19:22 | |
| kfox1111 | they need a solid base of k8s wrapped openstack building blocks. | 19:22 |
| kfox1111 | to swithc out the ones they already have running. | 19:22 |
| inc0 | it's data migration I'm worried about | 19:23 |
| srwilkers | afternoon | 19:24 |
| inc0 | and state migration like iptables rules | 19:24 |
| inc0 | hey Steve:) | 19:24 |
| kfox1111 | inc0: those are solvable. probably much more easiliy solvable then orcestrating greenfield. | 19:24 |
| inc0 | like...k8s will crap on iptables | 19:24 |
| inc0 | will that break what's there already | 19:24 |
| kfox1111 | if we don't have a db dump/restore option, we're hosed anyway. | 19:24 |
| openstackgerrit | Marcus Williams proposed openstack/kolla-ansible master: Add OpenDaylight role https://review.openstack.org/416367 | 19:24 |
| inc0 | yeah | 19:24 |
| kfox1111 | iptables is a pain, but managable. | 19:25 |
| inc0 | well we don't need tools build into kolla as long as we keep it documented tbh | 19:25 |
| inc0 | ops know how to dump/restore dbs | 19:25 |
| kfox1111 | I've had to hand roll iptables for pretty much every system I've deployed. | 19:25 |
| kfox1111 | yeah. | 19:25 |
| *** nathharp has joined #openstack-kolla | 19:26 | |
| kfox1111 | brownfield upgrades for snowflake systems should work today I think. | 19:26 |
| kfox1111 | but is easiest if there is a sideways migration path. | 19:26 |
| inc0 | yeah, in general | 19:26 |
| kfox1111 | which is more probable if we keep mitaka working. | 19:26 |
| inc0 | it's details that bites you | 19:26 |
| kfox1111 | so far, there's no real reason to drop it. its not making forward progres stop. | 19:27 |
| inc0 | so hold on, I'm not saying break it | 19:27 |
| kfox1111 | yeah. devils always in the details. :) | 19:27 |
| inc0 | let's just keep gates we have today for mitaka | 19:27 |
| kfox1111 | +1 | 19:27 |
| inc0 | and newton maybe | 19:27 |
| kfox1111 | +1 | 19:27 |
| inc0 | mitaka is closing to eol by the time we release | 19:27 |
| kfox1111 | once we get feature completeness in all the objects at 1.x, | 19:27 |
| inc0 | I'm saying not spend more time on it | 19:27 |
| kfox1111 | then we start forking off stable branches, and dropping suport in trunk for all but trunk. | 19:28 |
| inc0 | and do proper voting gates on tarballs on ocata+ | 19:28 |
| kfox1111 | I think the main effort left for 1.x in that space is, fluentd support, and prometheus support. | 19:28 |
| kfox1111 | I agree, once the fluentd/prometheus support is in. | 19:29 |
| kfox1111 | that should still work with older containers I think. | 19:29 |
| inc0 | yeah, we won't backport them to stable branches | 19:29 |
| inc0 | but you can install these few from dockerhub/tarballs | 19:29 |
| kfox1111 | yeah. thats kind of the point I'm trying to make. once we make stable branches, we don't backport to them. | 19:30 |
| kfox1111 | until then, I think we should not make stable branches until the k8s objects are feature complete. | 19:30 |
| *** manheim has joined #openstack-kolla | 19:30 | |
| inc0 | stable branches in kolla-k8s or kolla? | 19:30 |
| kfox1111 | kolla-k8s. | 19:30 |
| inc0 | in kolla-k8s agree | 19:30 |
| inc0 | on kolla proper, we already have stable | 19:30 |
| inc0 | when I'm talking about stable branches I'm talking about kolla proper | 19:31 |
| inc0 | anyway, gtg to airport, be back in few hrs | 19:31 |
| kfox1111 | right. | 19:31 |
| sbezverk | inc0: wait | 19:31 |
| kfox1111 | I'm only talking kolla-kubernetes. | 19:31 |
| sbezverk | could you please ack gate ps? | 19:31 |
| sbezverk | since kfox1111 and myself cannot +2 it now | 19:32 |
| sbezverk | zuul looks better now so there is a chance it will pass this time | 19:32 |
| kfox1111 | not sure if its needing more work or not... | 19:32 |
| kfox1111 | true. | 19:32 |
| kfox1111 | couldn't hurt to +2 it. | 19:32 |
| sbezverk | kfox1111: even if 3.X job still failing I think we need to merge it to unblock the gate | 19:33 |
| kfox1111 | sbezverk: I agree. but we may need to revert to get to that state. | 19:33 |
| kfox1111 | I think I got placement working, and tagged appropriately. but a slight chance it breaks something. | 19:34 |
| sbezverk | kfox1111: unless it is again binary vs source issue | 19:34 |
| *** manheim has quit IRC | 19:35 | |
| kfox1111 | that could be too. | 19:35 |
| kfox1111 | jsut not 100% sure I didn't typo an if in the ceph code. | 19:35 |
| kfox1111 | until Isee a gate run. | 19:35 |
| kfox1111 | worst case, we could disable placement in the nova config on 3.x | 19:36 |
| kfox1111 | then we would be operating like we use to. | 19:36 |
| mnaser | okay so | 19:38 |
| mnaser | an interesting problem regarding this (and binary) | 19:38 |
| mnaser | is the fact that packages are not pinned. | 19:38 |
| mnaser | building kolla tag 3.0.3 now and in 3 months will not result in the same image | 19:38 |
| mnaser | i dont think thats right | 19:38 |
| mnaser | and that would eliminate much of the problems regarding any potential packages issues breaking gate | 19:38 |
| kfox1111 | mnaser: yeah. thats a problem. | 19:39 |
| kfox1111 | but a huge amount of effort to keep up with the pins. | 19:39 |
| kfox1111 | I'd rather gate test individual builds constantlyh refreshing and pushing to dockerhub. | 19:39 |
| kfox1111 | and have users pull from that most of the time. | 19:39 |
| mnaser | crazy idea: drop the whole kolla versions and have images get tagged by the openstack version which they are running for example | 19:40 |
| kfox1111 | the exact artefacts used at the users site are then the same ones we validated worked with a full test cloud suite. | 19:40 |
| kfox1111 | mnaser: yeah. I've agued for that. | 19:40 |
| kfox1111 | but there are other versions too involved. | 19:40 |
| mnaser | but thats a lot of work and the base layers cause an issue | 19:40 |
| kfox1111 | like openssl version, etc. | 19:40 |
| mnaser | there are some simple pinning plugins for yum | 19:40 |
| mnaser | but the issue is keeping up with the releases | 19:40 |
| mnaser | we could have something similar to openstack proposal bot | 19:41 |
| kfox1111 | not really sure if it matters if the container is versioned off of mitaka = 2, or nova-api's mitaka version. | 19:41 |
| kfox1111 | the imporant bit is having a revision number that increments every time a change is made. | 19:41 |
| kfox1111 | so, centos-binary-nova-api:2.0.2-4 | 19:41 |
| kfox1111 | centos-binary-nova-api:2.0.2-5 | 19:41 |
| kfox1111 | etc. | 19:41 |
| kfox1111 | and we provide helm packages versioned similarly, that when a user upgrades, does a rolling upgrade to the new container. | 19:42 |
| mnaser | yeah that makes sense too | 19:42 |
| mnaser | its a very tricky subject | 19:42 |
| kfox1111 | yeah. | 19:42 |
| mnaser | but you still have to do some sort of pinning if you're going to let $user pin their own stuff | 19:42 |
| kfox1111 | yeah. we can support that toda with helm values. | 19:42 |
| * mnaser lives in kolla-ansible world | 19:43 | |
| kfox1111 | I'm more focused on making it so the user most of the ime doesn't have to. :) | 19:43 |
| kfox1111 | ah. | 19:43 |
| dmsimard | mnaser: I was told containers would fix all of my problems | 19:43 |
| kfox1111 | kolla-ansible might need a similar thing. | 19:43 |
| dmsimard | we live in a world of lies | 19:43 |
| kfox1111 | dmsimard: ;) | 19:43 |
| dmsimard | </friday> | 19:43 |
| mnaser | docker run openstack | 19:43 |
| mnaser | amirite | 19:43 |
| kfox1111 | heh. thats not a docker way to do things. ;) | 19:44 |
| dmsimard | right, it's docker run -it openstack /bin/bash | 19:45 |
| kfox1111 | no, I mean, thats a very very fat container. :) | 19:46 |
| kfox1111 | which is not how docker operates. :) | 19:46 |
| dmsimard | are you trying to say that openstack is fat | 19:46 |
| kfox1111 | yes. yes I am. :) | 19:46 |
| dmsimard | ¯\_(ツ)_/¯ | 19:46 |
| sbezverk | dmsimard: do you have this sequence as copy and paste or you memorized it ;) | 19:47 |
| sbezverk | ? | 19:47 |
| dmsimard | sbezverk: it's macro'ed :) | 19:47 |
| dmsimard | I have a few. | 19:47 |
| sbezverk | dmsimard: cool | 19:48 |
| dmsimard | <3 weechat | 19:48 |
| dmsimard | https://github.com/cmurphy/weechat-shrug/issues/1 | 19:48 |
| mnaser | lol | 19:49 |
| dmsimard | It's a bit more ugly in python :) https://github.com/cmurphy/weechat-shrug/blob/master/shrug.py#L10 | 19:49 |
| kfox1111 | logs.o.o still broken. :/ | 19:51 |
| mnaser | yeah they're running a big fsck on a giant 12tb volume kfox1111 | 19:51 |
| mnaser | not a fun time | 19:51 |
| sbezverk | mnaser: do you know any eta when it is going to be completed? | 19:52 |
| mnaser | sbezverk i believe 3 hours ago it was at around 89% | 19:52 |
| mnaser | so i think end of day today... dont think much more stuff will go in today | 19:52 |
| kfox1111 | wow. | 19:52 |
| sbezverk | mnaser: wow | 19:52 |
| mnaser | this fsck has been running since like | 19:53 |
| mnaser | 3am est | 19:53 |
| mnaser | so almost 12 hours now | 19:53 |
| kfox1111 | ok. I'll go work on non kolla-kubernetes thigns I guess. | 19:53 |
| mnaser | yeah.. its going to be a little while before things clear out | 19:53 |
| hawi | hi there. i have question regarding freezer - it gives me "Unable to get a job" error on ocata | 19:55 |
| -openstackstatus- NOTICE: lists.openstack.org will be offline from 20:00 to 23:00 UTC for planned upgrade maintenance | 19:58 | |
| *** mliima__ has quit IRC | 20:02 | |
| kfox1111 | busy day for infra. | 20:03 |
| sdake | poor folks | 20:05 |
| *** shashank_t_ has quit IRC | 20:06 | |
| *** shashank_t_ has joined #openstack-kolla | 20:07 | |
| *** rhallisey has quit IRC | 20:09 | |
| kfox1111 | so... kubeadm has gate tests. they just weren't working. :/ | 20:09 |
| *** shashank_t_ has quit IRC | 20:12 | |
| openstackgerrit | Sayantani Goswami proposed openstack/kolla master: [WIP] Added snap-telemetry container https://review.openstack.org/446164 | 20:15 |
| *** shashank_t_ has joined #openstack-kolla | 20:16 | |
| *** srwilkers has quit IRC | 20:22 | |
| *** rwallner has quit IRC | 20:24 | |
| *** rwallner has joined #openstack-kolla | 20:24 | |
| *** jtriley has quit IRC | 20:26 | |
| jascott1_ | ugh my halycon vagrant has decided to give up on life | 20:29 |
| kfox1111 | bummer. | 20:29 |
| *** Pavo has joined #openstack-kolla | 20:29 | |
| jascott1_ | yeah idk worked monday.. sometimes it goes out to lunch and i just do a destroy/up and am back in business. now k8s is complaining about 8080 not accessible | 20:31 |
| kfox1111 | did it go to k8s 1.6 on you? | 20:31 |
| jascott1_ | nah i checked and its at 1.5.1 | 20:31 |
| sbezverk | jascott1_: it must be cosmic radiation, kubernetes, zuul and now your vagrant ;) | 20:32 |
| jascott1_ | ^this checks out | 20:32 |
| kfox1111 | sbezverk: its been a horible week for system uptime. | 20:32 |
| kfox1111 | had some of my systems spontaniously explode too. | 20:32 |
| kfox1111 | not sure whats going on. :) | 20:32 |
| kfox1111 | planitary alinement, gremins, etc. :) | 20:33 |
| sbezverk | :) | 20:33 |
| kfox1111 | I did notice jupiter's nice and brite in the sky right now. :) | 20:33 |
| *** rwallner has quit IRC | 20:34 | |
| openstackgerrit | Sayantani Goswami proposed openstack/kolla-ansible master: [WIP] Added playbooks for snap deployment https://review.openstack.org/446165 | 20:37 |
| jascott1_ | all dem monolifs | 20:41 |
| kfox1111 | :) | 20:41 |
| jascott1_ | btw the golang class was great, hope I retained enough og it | 20:43 |
| jascott1_ | og/of | 20:43 |
| sbezverk | kfox1111: wow, you can see planets during the day time?!?!? | 20:50 |
| sbezverk | jascott1_: which one, I mean class? | 20:50 |
| *** Pavo has quit IRC | 20:51 | |
| jascott1_ | sbezverk https://github.com/ardanlabs/gotraining | 20:51 |
| jascott1_ | Bill Kennedy that wrote "Go in Action" | 20:51 |
| jascott1_ | what I liked is that it wasn't a run thru of the syntax (that you can get anywhere) but more of a 'why they did it that way' | 20:53 |
| sdake | jascott1_ nice - i looked into go trianing at cisco - 15k per head | 20:53 |
| sdake | jascott1_ needless to say, I decided to pass | 20:53 |
| jascott1_ | sdake you mean 1.5K ? | 20:53 |
| sdake | no i mean 15k | 20:54 |
| jascott1_ | our was approx 2K | 20:54 |
| jascott1_ | per head | 20:54 |
| sdake | yup | 20:54 |
| sdake | 15k is rediculous | 20:54 |
| sbezverk | jascott1_: I would go for 2k training | 20:54 |
| sdake | i'm in the wrong business | 20:54 |
| jascott1_ | weird.. maybe they have minimum | 20:54 |
| jascott1_ | sdake i hear ya | 20:54 |
| sdake | sbezverk buying 3 of these: http://www.cisco.com/c/dam/en/us/products/collateral/servers-unified-computing/ucs-c-series-rack-servers/c220m4-lff-spec-sheet.pdf | 20:55 |
| sdake | thoughts? | 20:55 |
| sbezverk | 15k training was probably held in Hawaii or Bora-Bora | 20:55 |
| jascott1_ | its some nice math when you look around and count the students | 20:55 |
| jascott1_ | in your bedroom! stay in your pjs! | 20:55 |
| sdake | jascott1_ nobody really understands how expensive it is to run a business | 20:55 |
| sdake | insurance, attorneys, accountants, cleaning people, the list goes on | 20:56 |
| jascott1_ | agree | 20:56 |
| sbezverk | sdake: I have two of these very happy | 20:56 |
| sdake | sbezverk at your house or work provided | 20:56 |
| sbezverk | at my house work rpovided | 20:56 |
| sbezverk | provided | 20:56 |
| sdake | how did you hook that up?: ) | 20:56 |
| openstackgerrit | Sayantani Goswami proposed openstack/kolla master: Install InfluxDB Plugin in Grafana https://review.openstack.org/446624 | 20:57 |
| *** Pavo has joined #openstack-kolla | 20:57 | |
| sdake | what i want is an employee discount on that gear | 20:57 |
| sbezverk | I could run about 15 servers lab, but then my electricity bill would go through the roof | 20:57 |
| sdake | since, you know, I'm an EMPLOYEE | 20:57 |
| sbezverk | so I run 1 or 2 at the time | 20:57 |
| *** rwallner has joined #openstack-kolla | 20:59 | |
| sdake | sbezverk do you have a 2557 vic? | 20:59 |
| openstackgerrit | Sayantani Goswami proposed openstack/kolla master: Install InfluxDB Plugin in Grafana https://review.openstack.org/446624 | 20:59 |
| sbezverk | hmmm, never heard this part number | 20:59 |
| sdake | its a nic on the board you mount | 21:00 |
| sdake | iLOM | 21:00 |
| sdake | have to buy this stuff myself - time to replace the minidells | 21:00 |
| sdake | or atleast add to them | 21:00 |
| jascott1_ | dang helm plugin 'template' doesnt really show computed values | 21:00 |
| sdake | sbezverk how is their volume level | 21:00 |
| sbezverk | sdake: I keep them in the basement | 21:01 |
| sdake | sbezverk page 3 of that spec sheet - there are 8 nics on the thing | 21:01 |
| sdake | what are all those nics for? | 21:01 |
| sdake | nm i just got to page 5:) | 21:02 |
| *** rwallner has quit IRC | 21:03 | |
| sbezverk | you got vic wrong | 21:03 |
| *** nathharp has quit IRC | 21:03 | |
| sbezverk | vin usually is 1227 or 1387 | 21:03 |
| sbezverk | vic I meant | 21:03 |
| sdake | sbezverk sorry 1227 is what i meant | 21:04 |
| sdake | i apparently need to get a RJ45 version to fit in with my 100$/port switch | 21:04 |
| sdake | cisco switches (nexus 9k) are 1k/port | 21:04 |
| sbezverk | you should be careful with vic models | 21:04 |
| sdake | please expand :) | 21:04 |
| sbezverk | as they usually | 21:04 |
| sbezverk | use sfp for connectors | 21:05 |
| sdake | yup i know | 21:05 |
| *** shashank_t_ has quit IRC | 21:05 | |
| sdake | i found an rj45 version | 21:05 |
| sdake | its 900 bucks | 21:05 |
| sdake | the server is 1800 | 21:05 |
| sdake | rediculous the nic is 1/3rd the price of the server | 21:05 |
| sdake | so how is the noise level when your in the basement? | 21:05 |
| sdake | loud or quiet? | 21:06 |
| sdake | my minidells are super loud | 21:06 |
| sbezverk | it is smart card, you can slice it on multiple logical nics | 21:06 |
| sdake | yup i know | 21:06 |
| sdake | up to 256 nics or hba | 21:06 |
| sbezverk | I can still hear it | 21:06 |
| sbezverk | but very low | 21:06 |
| sdake | i can hear mine from the other side of the house | 21:06 |
| sdake | and have no basement | 21:06 |
| sdake | minidell is tight on space | 21:06 |
| sdake | so hae to turn fans on max so my intel nics dont overheat | 21:07 |
| sdake | i think 2 of my servers are dead | 21:07 |
| sdake | do you run cimc3.0? | 21:07 |
| sbezverk | these are pizza box rack width, 1U and full lengthj | 21:07 |
| sbezverk | yes | 21:07 |
| sbezverk | use 3.0 | 21:07 |
| *** gfidente has quit IRC | 21:07 | |
| sdake | ya 2.z has the terrible interface | 21:08 |
| sdake | for the console | 21:08 |
| sdake | i've found this one: http://www.provantage.com/cisco-systems-ucs-spr-c220m4-bs1~7CSC8AUR.htm | 21:08 |
| sdake | which is SFF | 21:09 |
| sdake | at 1700 | 21:09 |
| sdake | trouble finding a LFF | 21:09 |
| *** Pavo has quit IRC | 21:10 | |
| sbezverk | hm, I thought sff is more difficult to find | 21:10 |
| sdake | this is a good deal http://www.provantage.com/cisco-systems-ucs-spr-c220m4-bs2~7CSC8AUN.htm | 21:10 |
| sdake | wtb in lff | 21:11 |
| sbezverk | 8 cores?!?! per processor, that sucks | 21:12 |
| sdake | ya - 16 hyperthreads = 32 cpus total | 21:13 |
| sdake | plenty for me | 21:14 |
| sdake | (for persona luse) | 21:14 |
| openstackgerrit | Mohammed Naser proposed openstack/kolla stable/newton: Fix magnum [DEFAULT] configuration. https://review.openstack.org/452326 | 21:14 |
| sdake | sbezverk i am paying out of pocket | 21:14 |
| sdake | I have a lab internally I can use already | 21:14 |
| *** pramodrj07 has joined #openstack-kolla | 21:15 | |
| *** sayantan_ has quit IRC | 21:18 | |
| sbezverk | sdake: well you are rich man, you can afford it ;) I cannot, I need to rely on what my employer can give me :) | 21:21 |
| *** esharao has quit IRC | 21:21 | |
| sdake | sbezverk im just as broke as the rest of america :) | 21:21 |
| sdake | 31" deep | 21:22 |
| * sdake groans | 21:22 | |
| sbezverk | it is nice server, but not great for house interrior design | 21:24 |
| sbezverk | some people might not appreciate it is technical beauty | 21:24 |
| *** goldyfruit has quit IRC | 21:26 | |
| sdake | sbezverk IDC :) | 21:27 |
| sdake | sbezverk re the SFF vs the LFF | 21:27 |
| sdake | sbezverk I can't seem to find an LFF at an acceptable price | 21:27 |
| sdake | or an LFF at all | 21:28 |
| sdake | I guess my SSDs are all SFF | 21:28 |
| *** schwicht has quit IRC | 21:28 | |
| sdake | would you go with SFF or LFF? | 21:28 |
| sbezverk | lff is more noisy | 21:31 |
| sbezverk | but you can get more disk space with lff | 21:31 |
| sbezverk | 4tb per lff | 21:31 |
| sbezverk | sff I think can go up 2 tb but they are very expensive | 21:32 |
| *** shashank_t_ has joined #openstack-kolla | 21:38 | |
| *** nathharp has joined #openstack-kolla | 21:40 | |
| *** nathharp has quit IRC | 21:41 | |
| Mech422 | sbezverk: if you want bargin stuff - I'm happy with my HP blades, and mostly happy with my Dell C6100 blades | 21:42 |
| Mech422 | sbezverk: both clock in around $200/node for 16-32G RAM, and I think dual quad cores | 21:42 |
| Mech422 | sbezverk: both are noiser then jet engine though | 21:43 |
| sbezverk | Mech422: nah I am ok with my cisco ucs boxes, I just love them ;) | 21:43 |
| Mech422 | sbezverk: ahh - I thought you guys were looking for new gear... | 21:43 |
| sbezverk | Mech422: that was sdake | 21:43 |
| Mech422 | sbezverk: sdake likes his 'quiet' stuff - I already tried to corrupt him :-) | 21:43 |
| Mech422 | sbezverk: thing thats annoying me now is the cost of KVM cables | 21:44 |
| Mech422 | sbezverk: you can get some NICE deals on Raritan IP KVMs on ebay atm | 21:44 |
| Mech422 | but the cables cost a fortune | 21:44 |
| sdake | Mech422 with these servers you dont need any kvm cable at all | 21:44 |
| sdake | Mech422 the console is completey virtual | 21:44 |
| sbezverk | sdake: you do, but they come with it | 21:45 |
| sdake | htmlv5 | 21:45 |
| sdake | sbezverk you can't run the server without a keyboard/mouse/etc? | 21:45 |
| sbezverk | sdake: you can, but you need first to setup IP for cimc access, after that you are ok | 21:46 |
| sdake | ok i can live with that | 21:46 |
| sdake | does cimc ip support DHCP? | 21:46 |
| Mech422 | ahh - the IPMI consoles on mine are either crap (Dell) or too expensive to license (HP) | 21:46 |
| sbezverk | sdake: most likely, I always use fixed ip | 21:46 |
| sbezverk | sdake: how do you find out which ip it took ;) ?? | 21:47 |
| sdake | sbezverk dns resolution via eero | 21:47 |
| sbezverk | unles you start browing dhcp server logs | 21:47 |
| sdake | i can assign a name to an ip address | 21:47 |
| sdake | and then telnet xyz and my eero dns resolves for me to hte correct ip | 21:47 |
| sdake | sbezverk does the vic mandate a cisco switch? | 21:48 |
| sdake | or will it work with any switch | 21:48 |
| sbezverk | I prefer to connect to the server, setup ip and then no headache | 21:48 |
| sbezverk | sdake: nope but you need trunking capability on the switch | 21:49 |
| sbezverk | so what ever switch you use it must be 802.1q compatible | 21:49 |
| -openstackstatus- NOTICE: The upgrade maintenance for lists.openstack.org has been completed and it is back online. | 21:50 | |
| *** schwicht has joined #openstack-kolla | 21:52 | |
| sdake | sbezverk this is what I've got: https://www.amazon.com/NETGEAR-ProSAFE-10-Gigabit-Ethernet-XS708E-200NES/dp/B01ELW0QM4/ref=sr_1_7?ie=UTF8&qid=1490997028&sr=8-7&keywords=netgear%2B8%2Bport%2B10%2Bgigabit%2Bswitch&th=1 | 21:52 |
| sdake | Mind you, I am not using this switch in a commercial production setting, but in a home setting with a probably overly complicated network topology. That said, it replaced a much larger switch with extremely noisy fans with a unit that's barely bigger than the ports themselves, and draws a nominal 6 watts in normal use. All without losing the features I had to have, 802.3ad, 802.1q, etc. | 21:53 |
| sdake | in one of hte reviews | 21:53 |
| sdake | so theoretially should work | 21:54 |
| sbezverk | yeah, looks like it should work since 802.1q is mentioned in specs | 21:54 |
| sdake | annoying windows only utility | 21:58 |
| sdake | i have used this tool before | 21:58 |
| mnaser | man | 22:13 |
| mnaser | this feels so dirty | 22:13 |
| mnaser | magnum had a missing dependency in ocata | 22:13 |
| mnaser | which openstack/rdo addressed | 22:13 |
| mnaser | and all im doing is running kolla-build and im getting a fixed image. but i made no changes to kolla. | 22:13 |
| mnaser | tagging them 4.0.0.1 but... this cant be right :< | 22:13 |
| sbezverk | jascott1_: still around? | 22:14 |
| dmsimard | mnaser: so the way I see that | 22:14 |
| dmsimard | mnaser: is that 4.0.0.1 is the kolla version, not the magnum version | 22:14 |
| jascott1_ | sbezverk yes sir | 22:14 |
| dmsimard | mnaser: the magnum version is probably like 1a2b3c | 22:15 |
| dmsimard | you can call your container 4.0.0.1-1a2b3c if you want | 22:15 |
| mnaser | dmsimard yeah. in build time, we should detect the installed version and tag accordingly (but then it introduces the "how do i detect what to install" in kolla-ansible) | 22:15 |
| sbezverk | jascott1_: strange problem, getting this error : ./main.go:18: undefined: ssh.FixedHostKey | 22:15 |
| dmsimard | FWIW in RDO (trunk) I purposefully label the hash of the repository the container was built with | 22:16 |
| sbezverk | I do import "golang.org/x/crypto/ssh" | 22:16 |
| sbezverk | where the doc says it's defined | 22:16 |
| sbezverk | any clues/suggestions? | 22:16 |
| jascott1_ | hmm | 22:16 |
| mnaser | dmsimard yeah.. this has to be addressed. 4.0.0 bits of kolla should always give you the same bits | 22:17 |
| jascott1_ | could you pastebin? | 22:17 |
| sbezverk | the error? | 22:17 |
| mnaser | not different images depending on the day/time its built | 22:17 |
| dmsimard | mnaser: I see where you're going but I'd argue the contrary | 22:17 |
| dmsimard | mnaser: 4.0.0 is the version of the Kolla docker files | 22:17 |
| dmsimard | not the resulting version of the docker images | 22:17 |
| mnaser | dmsimard fair, then we shouldn't be tagging them in dockerhub as 4.0.0 | 22:18 |
| dmsimard | point taken | 22:18 |
| jascott1_ | sbezverk dump the ssh to stdout to see whats in it | 22:18 |
| dmsimard | the version of the docker image != the version of the docker file | 22:18 |
| mnaser | i agree with that 100% dmsimard .. so i guess how to decide whats the 'appropriate' tag | 22:18 |
| dmsimard | the version of the docker image should be a derivative of the git sha1 (source) or rpm/dpkg version (binary) + kolla version | 22:18 |
| dmsimard | mnaser: probably something something pbr | 22:19 |
| jascott1_ | sbezverk make sure you are looking at right docs for package, ive been looking at the wrong ones before with same name but different namespace | 22:19 |
| jascott1_ | and you get 'wheres my method? etc' | 22:19 |
| mnaser | dmsimard and then the second (maybe not so fun part) is how to get kolla-ansible and co to find out what tag to use | 22:20 |
| sbezverk | jascott1_: http://paste.openstack.org/show/605112/ | 22:20 |
| sbezverk | jascott1_: https://godoc.org/golang.org/x/crypto/ssh#FixedHostKey | 22:20 |
| dmsimard | mnaser: tag them latest and 4.0.0-1a2b3c | 22:20 |
| dmsimard | people who care use the pinned tag, people who don't use latest | 22:21 |
| *** sayantan_ has joined #openstack-kolla | 22:21 | |
| dmsimard | not unlike pip install ansible=2.2.2.0 vs pip install ansible | 22:21 |
| *** rwellum has quit IRC | 22:21 | |
| *** signed8b_ is now known as signed8bit_Zzz | 22:21 | |
| *** sayantan_ has quit IRC | 22:22 | |
| jascott1_ | sbezverk its a func, you are calling it? | 22:22 |
| jascott1_ | its buried in the docs amongst types | 22:22 |
| dmsimard | but anyway I'm not going to pretend I'm knowledgeable about this, I'm bad at docker :) | 22:22 |
| *** sayantan_ has joined #openstack-kolla | 22:22 | |
| mnaser | dmsimard that's actually good insight, i am going to give it a bit more thought | 22:22 |
| dmsimard | mnaser: but that's a bit what we're (trying) to do in RDO | 22:22 |
| mnaser | no this is really useful, the decoupling and the multiple tags to identify different releases, allowing a user to pin to specific images to specific releases (if they want) or stick to latest of a branch (master/ocata/newton) | 22:23 |
| sbezverk | jascott1_: it is called as callback HostKeyCallback: ssh.FixedHostKey(hostKey), | 22:23 |
| dmsimard | mnaser: so we have delorean trunk repository hashes, right, https://trunk.rdoproject.org/centos7/ed/de/edde55febc7ec2284e1f1475964da07d1ac127ab_1c6d6691 | 22:23 |
| jascott1_ | sbeverk log that ssh object | 22:23 |
| dmsimard | mnaser: when we build the "latest" images, we recover that hash -- and the image is tagged with that hash and latest | 22:23 |
| jascott1_ | %v | 22:23 |
| dmsimard | mnaser: https://hub.docker.com/r/rdoproject/centos-binary-swift-proxy-server/tags/ | 22:24 |
| dmsimard | (Don't use those :D) | 22:24 |
| sbezverk | jascott1_: do you mean, commented out whole thing and just print out element of the object? | 22:24 |
| mnaser | dmsimard thats an interesting approach, maybe having different streams for branches (i.e. newton/ocata) and that seems like an ideal way of doing things | 22:25 |
| mnaser | at least in $rdo world | 22:25 |
| dmsimard | mnaser: we also add labels inside the container https://gist.github.com/dmsimard/585e8af3a676db666c333a88e8f07d71#file-patched_overrides-j2-L35-L36 | 22:25 |
| jascott1_ | import ( "log") then log.Fatal(ssh) | 22:25 |
| jascott1_ | if that method isnt there then what have you got? | 22:25 |
| *** signed8bit_Zzz is now known as signed8b_ | 22:26 | |
| *** schwicht has quit IRC | 22:26 | |
| dmsimard | mnaser: gotta go o/ | 22:26 |
| jascott1_ | oh sorry | 22:26 |
| mnaser | later man, be safe out there, thanks for the talk dmsimard | 22:26 |
| mnaser | snow tonight :( | 22:26 |
| sbezverk | mnaser: where are you located? | 22:28 |
| mnaser | montreal canada sbezverk | 22:28 |
| *** fooliouno has quit IRC | 22:28 | |
| *** williamwang has quit IRC | 22:28 | |
| sbezverk | mnaser: me too, but I do not see any snow | 22:29 |
| sbezverk | at least yet | 22:29 |
| mnaser | sbezverk oh no way | 22:29 |
| mnaser | i dont either but it looks like it's on its way | 22:29 |
| mnaser | pretty cool, didnt know this whole time you lived here :-P | 22:29 |
| inc0 | back | 22:29 |
| *** shashank_t_ has quit IRC | 22:29 | |
| sbezverk | mnaser: right, me too | 22:30 |
| *** shashank_t_ has joined #openstack-kolla | 22:30 | |
| mnaser | sbezverk were you at the openstack canada day by any chance? | 22:30 |
| sbezverk | mnaser: nope, by some reason I do not get any notification about local events | 22:31 |
| mnaser | oh bummer, i'll ping you if there's one coming up soon! | 22:31 |
| mnaser | some people expressed interest in hearing a talk about how we migrated to kolla so i might be doing a talk about that next meetu p | 22:32 |
| sbezverk | mnaser: sure, would appreciate it | 22:32 |
| *** bjolo has quit IRC | 22:33 | |
| *** new2kolla_ has quit IRC | 22:33 | |
| *** shashank_t_ has quit IRC | 22:35 | |
| jascott1_ | sbezverk kfox1111 fyi they have changed some kubeadm params in 1.6 | 22:42 |
| *** signed8b_ is now known as signed8bit_Zzz | 22:42 | |
| *** adrian_otto has quit IRC | 22:43 | |
| sbezverk | jascott1_: yeah we know, we have already addressed it in new gate ps | 22:43 |
| jascott1_ | halycon can get mismatched apparently | 22:44 |
| sbezverk | we just cannot merge it because of dying zuul | 22:44 |
| jascott1_ | i think its installing latest kubeadmin and fixed k8s | 22:44 |
| *** adrian_otto has joined #openstack-kolla | 22:52 | |
| jascott1_ | sbezverk was that indeed the package you were looking for? :) | 23:06 |
| sbezverk | jascott1_: if you mean ssh, then yes | 23:07 |
| jascott1_ | where was your method? | 23:07 |
| jascott1_ | er.. Func | 23:07 |
| *** goldyfruit has joined #openstack-kolla | 23:08 | |
| kfox1111 | sbezverk: now as in these days, not now as in this second. :) | 23:14 |
| *** unicell has quit IRC | 23:14 | |
| *** adrian_otto has quit IRC | 23:15 | |
| sbezverk | jascott1_: I figured out | 23:15 |
| kfox1111 | logs back yet? | 23:16 |
| sbezverk | that parameter exists only in go 1.8 | 23:16 |
| jascott1_ | ah | 23:16 |
| sbezverk | but I use go 1.7.5 here it is not defined | 23:16 |
| jascott1_ | 1.8 is stable iirc | 23:16 |
| jascott1_ | yeah file that next to what i said about versions of the docs :) | 23:17 |
| jascott1_ | the last two people that asked me a go related thing ended up being doc version mismatch related | 23:18 |
| *** adrian_otto has joined #openstack-kolla | 23:18 | |
| *** manheim has joined #openstack-kolla | 23:22 | |
| *** adrian_otto has quit IRC | 23:25 | |
| *** manheim has quit IRC | 23:26 | |
| *** sayanta__ has joined #openstack-kolla | 23:30 | |
| *** sayantan_ has quit IRC | 23:30 | |
| *** caoyuan has joined #openstack-kolla | 23:39 | |
| *** deadnull has quit IRC | 23:44 | |
| jascott1_ | sbezverk know if 1.6 requires changes to CNI version? | 23:45 |
| jascott1_ | sbezverk haha i found the issue on github and you were already there sharing the workaround | 23:47 |
| kfox1111 | jascott1_: I believe so. | 23:48 |
| kfox1111 | theres 0.5.x in the repo. | 23:48 |
| *** unicell has joined #openstack-kolla | 23:52 | |
| kfox1111 | who is this coeki person? https://github.com/kubernetes/kubernetes/issues/43815#issuecomment-290863330 | 23:53 |
| jascott1_ | oh see you are on the thread there... | 23:53 |
| jascott1_ | idk | 23:53 |
| kfox1111 | yeah. github doesn't have much to say. drive by reviewer I guess. | 23:54 |
| kfox1111 | I'm going to just ignore him... The clue bat's not working. :) | 23:54 |
| jascott1_ | yeah i dont get his last comment | 23:55 |
| jascott1_ | well thought i had halycon patched up until i hit this :/ | 23:55 |
| jascott1_ | sounds like music and beer time to me | 23:55 |
| kfox1111 | :) | 23:55 |
| kfox1111 | sounds like logs.o.o is a ways out still. | 23:56 |
| kfox1111 | music and beer it is. :) | 23:57 |
| jascott1_ | lol if you want to make the 3hr trek, beer is on me :) | 23:57 |
| kfox1111 | yeah... thats a costly free beer. :) | 23:58 |
| kfox1111 | if it wasn't for the 3 hours though, would be happy to hang out. | 23:58 |
| *** goldyfruit has quit IRC | 23:59 | |
| jascott1_ | no doubt next time you are in town we cant paint it any colour you like | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!