Thursday, 2018-10-04

« Wednesday, 2018-10-03 Index Friday, 2018-10-05 »
*** imdigitaljim has quit IRC00:18
*** imdigitaljim has joined #openstack-containers00:26
*** dave-mccowan has quit IRC00:52
*** hongbin has joined #openstack-containers01:55
*** Bhujay has joined #openstack-containers02:24
*** ricolin has joined #openstack-containers02:28
*** ramishra has joined #openstack-containers02:45
*** Bhujay has quit IRC03:03
*** ricolin has quit IRC03:08
*** udesale has joined #openstack-containers03:14
*** hongbin has quit IRC03:27
*** ykarel|away has joined #openstack-containers04:02
*** cbrumm has quit IRC04:07
*** imdigitaljim has quit IRC04:15
*** ianychoi_ has joined #openstack-containers04:15
*** ianychoi has quit IRC04:17
*** udesale has quit IRC05:09
*** udesale has joined #openstack-containers05:15
*** ykarel|away is now known as ykarel05:23
*** udesale has quit IRC05:53
*** udesale has joined #openstack-containers05:58
*** fghaas has joined #openstack-containers06:28
*** serlex has joined #openstack-containers06:30
*** rcernin has quit IRC06:38
*** rcernin has joined #openstack-containers06:38
*** pcaruana has joined #openstack-containers06:40
*** jaewook_oh has joined #openstack-containers06:54
*** ttsiouts has joined #openstack-containers06:59
*** rcernin has quit IRC07:10
*** ttsiouts has quit IRC07:16
*** ykarel_ has joined #openstack-containers07:24
*** ykarel has quit IRC07:26
*** mattgo has joined #openstack-containers07:27
*** ykarel__ has joined #openstack-containers07:28
openstackgerritFeilong Wang proposed openstack/magnum-tempest-plugin master: Support k8s testing  https://review.openstack.org/60432307:28
*** ykarel_ has quit IRC07:30
*** serlex has quit IRC07:30
*** udesale has quit IRC07:30
*** ykarel_ has joined #openstack-containers07:41
*** ykarel__ has quit IRC07:44
*** ykarel__ has joined #openstack-containers07:46
*** ykarel__ is now known as ykarel07:46
*** ykarel has quit IRC07:47
*** ykarel_ has quit IRC07:48
*** udesale has joined #openstack-containers07:53
*** ykarel has joined #openstack-containers07:54
*** pcaruana has quit IRC07:55
*** pcaruana has joined #openstack-containers07:57
*** ttsiouts has joined #openstack-containers08:00
openstackgerritOpenStack Proposal Bot proposed openstack/magnum-ui master: Imported Translations from Zanata  https://review.openstack.org/60783408:02
*** ykarel_ has joined #openstack-containers08:12
*** ykarel has quit IRC08:15
*** ykarel__ has joined #openstack-containers08:20
*** ykarel_ has quit IRC08:23
*** ykarel_ has joined #openstack-containers08:23
openstackgerritMerged openstack/magnum-tempest-plugin master: fix typo  https://review.openstack.org/59799308:23
*** ykarel__ has quit IRC08:26
*** serlex has joined #openstack-containers08:28
*** ykarel__ has joined #openstack-containers08:45
*** flwang1 has joined #openstack-containers08:45
*** udesale has quit IRC08:46
*** ykarel_ has quit IRC08:48
*** ttsiouts has quit IRC08:49
*** ykarel_ has joined #openstack-containers08:49
*** udesale has joined #openstack-containers08:49
*** ykarel__ has quit IRC08:51
*** ttsiouts has joined #openstack-containers08:59
*** ttsiouts has quit IRC09:04
*** ykarel_ has quit IRC09:07
*** ykarel_ has joined #openstack-containers09:07
*** ttsiouts has joined #openstack-containers09:08
*** ykarel__ has joined #openstack-containers09:10
*** ykarel_ has quit IRC09:13
*** ykarel has joined #openstack-containers09:18
*** ykarel__ has quit IRC09:19
*** salmankhan has joined #openstack-containers09:24
flwang1any Blizzard people around?09:30
*** ykarel has quit IRC09:32
*** ykarel has joined #openstack-containers09:32
*** ykarel_ has joined #openstack-containers09:35
*** ykarel has quit IRC09:38
*** ykarel_ is now known as ykarel09:43
*** Bhujay has joined #openstack-containers09:48
*** udesale has quit IRC10:04
*** Bhujay has quit IRC10:06
*** mattgo has quit IRC10:11
*** udesale has joined #openstack-containers10:13
*** ttsiouts has quit IRC10:21
*** ricolin has joined #openstack-containers10:29
*** ttsiouts has joined #openstack-containers10:31
*** udesale has quit IRC10:39
*** udesale has joined #openstack-containers10:40
*** mattgo has joined #openstack-containers10:52
*** pcaruana has quit IRC11:02
*** pcaruana has joined #openstack-containers11:02
*** janki has joined #openstack-containers11:06
*** ttsiouts has quit IRC11:32
*** slagle has quit IRC11:51
*** jaewook_oh_ has joined #openstack-containers12:03
*** jaewook_oh has quit IRC12:03
*** jaewook_oh_ is now known as jaewook_oh12:03
*** ttsiouts has joined #openstack-containers12:07
*** jaewook_oh has quit IRC12:44
*** fghaas has quit IRC12:45
*** slagle has joined #openstack-containers12:47
*** slagle has quit IRC13:22
*** ykarel_ has joined #openstack-containers13:36
*** ykarel has quit IRC13:38
*** slagle has joined #openstack-containers13:41
*** fghaas has joined #openstack-containers13:42
openstackgerritTheodoros Tsioutsias proposed openstack/magnum-specs master: Introduce magnum nodegroups  https://review.openstack.org/60736313:49
flwang1eandersson: ping13:49
*** pcaruana has quit IRC13:50
*** ykarel_ is now known as ykarel13:50
*** hongbin has joined #openstack-containers14:17
*** Bhujay has joined #openstack-containers14:28
*** fghaas has quit IRC14:29
*** ykarel is now known as ykarel|afk14:32
*** itlinux has quit IRC14:48
*** jchhatbar has joined #openstack-containers14:50
*** janki has quit IRC14:53
colin-flwang1 o/15:06
colin-need review?15:07
*** ttsiouts has quit IRC15:11
*** slagle has quit IRC15:25
*** udesale has quit IRC15:36
*** mattgo has quit IRC15:37
*** jchhatbar has quit IRC15:46
*** ykarel|afk is now known as ykarel15:52
*** Bhujay has quit IRC15:55
*** itlinux has joined #openstack-containers15:57
*** ykarel_ has joined #openstack-containers16:18
*** ykarel_ is now known as ykarel|away16:20
*** ykarel has quit IRC16:20
*** slagle has joined #openstack-containers16:20
*** Bhujay has joined #openstack-containers16:34
*** cbrumm has joined #openstack-containers16:55
*** ykarel|away has quit IRC16:58
*** Bhujay has quit IRC17:01
*** salmankhan has quit IRC17:04
*** ramishra has quit IRC17:09
*** pcaruana has joined #openstack-containers17:17
*** Bhujay has joined #openstack-containers17:30
*** ricolin has quit IRC17:44
colby_Hey Guys,17:56
colby_We are running pike version of magnum. If we tell magnum to build kubernetes cluster without floating ip it fails. If we tell it to do floating ip it gives all nodes floating ips and the minions have security group rules allowing all traffic to those nodes (not very secure). Is there a way to fix this so only the master gets floating ip?17:58
*** ykarel|away has joined #openstack-containers17:59
*** pcaruana has quit IRC18:01
*** ykarel_ has joined #openstack-containers18:04
*** ykarel|away has quit IRC18:07
*** Bhujay has quit IRC18:20
*** imdigitaljim has joined #openstack-containers18:36
imdigitaljimstrigazi: you here?18:36
imdigitaljimsidenote: im constantly dc'd from IRC lately so hit me on email if you need something btw18:38
strigaziimdigitaljim: here18:39
imdigitaljimhey18:41
strigazicolby_: no, sorry, this is option is not possible. either all nodes will have fips or none. I know it is not  great but it is like this atm18:41
imdigitaljimso i was looking at your heat-container ssh PR18:41
imdigitaljimit looks pretty good but one thing i wanted to see if i missed18:41
imdigitaljimdo you enforce on the sshd_config to enable rootlogin18:41
imdigitaljimyou might want to have a sed for PermitRootLogin without-password18:41
colby_strigazi: is it possible to change the security group rules then. Seem like allowing all traffic to the minion fips is not ideal.18:42
strigaziI generate the ssh key for root in srv magnum. I can take a look in that option too18:43
strigaziimdigitaljim: ^^18:43
strigazicolby_: you can change them with the neutron API after cluster creation18:43
strigazicolby_: no other option is in place atm18:44
strigazicolby_: the optimal option is what you proposed18:44
strigazicolby_: I have kickstarted the implementation but then I had too many on my plate and since at CERN we didn;t need it it went to the backlog18:44
colby_ok thanks for the info18:45
imdigitaljimwell if sshd_config has PermitRootLogin no18:45
imdigitaljimthe ssh call from heat didnt work for me18:46
imdigitaljimheat-container*18:46
imdigitaljim(with the same code as your PR)18:46
strigazihmm18:46
strigaziI'll have a look18:46
imdigitaljimit gets permission denied18:46
imdigitaljimbut adding that to sshd_ works like a charm18:46
imdigitaljimand its a rather clean solution that the problem18:46
imdigitaljimso +1 on that18:47
strigazican you leave a comment in gerrit?18:47
imdigitaljimyeah i was in the middle of doing that as we speak18:47
strigaziimdigitaljim: btw Theodoros from our team update the nodegroups spec.18:47
strigaziimdigitaljim: thx18:47
imdigitaljimok thanks!18:48
flwang1imdigitaljim: did you see my email?18:50
imdigitaljimoh no18:50
imdigitaljimjust checked18:50
imdigitaljimreading..18:50
flwang1imdigitaljim: What's the version of your coreDNS and Calico? With the default         versions in Magnum, CoreDNS 1.0.1 and Calico 2.6.7, on k8s         v1.11.218:50
flwang1we got http://paste.openstack.org/show/731482/18:51
flwang1the dns is not working18:51
flwang1and did you do any special change for the coredns or calico config?18:51
imdigitaljimetcd 3.3.9 calico v3.2.1 coredns 1.2.018:52
imdigitaljimcalico was primarily update the configmap to the newer one18:53
imdigitaljimfrom calico's page18:53
imdigitaljimvery minor changes18:53
imdigitaljimcoredns i dont think had any changes but it was missing an important feature18:53
imdigitaljimwhich might be part of your problem?18:53
imdigitaljim  Corefile: |18:54
imdigitaljim    .:53 {18:54
imdigitaljim        errors18:54
imdigitaljim        log stdout18:54
imdigitaljim        health18:54
imdigitaljim        kubernetes ${DNS_CLUSTER_DOMAIN} ${PORTAL_NETWORK_CIDR} ${PODS_NETWORK_CIDR} {18:54
imdigitaljim            pods verified18:54
imdigitaljim            upstream18:54
imdigitaljim        }18:54
imdigitaljim        proxy . /etc/resolv.conf18:54
imdigitaljim        cache 3018:54
imdigitaljim    }18:54
imdigitaljimthe upstream part of the kubernetes plugin18:54
imdigitaljimoh and k8s version 1.11.3/1.12.0 currently but by next week just 1.12.018:54
strigaziimdigitaljim:  I think this is what we have already for coredns18:54
imdigitaljimwe trickle people over as we upgrade18:54
imdigitaljimstrigazi: oh was it added since i had a pull18:55
flwang1imdigitaljim: it has been added, i mean  ${DNS_CLUSTER_DOMAIN} ${PORTAL_NETWORK_CIDR} ${PODS_NETWORK_CIDR} {18:55
imdigitaljimhttps://github.com/openstack/magnum/blob/master/magnum/drivers/common/templates/kubernetes/fragments/core-dns-service.sh#L6618:55
imdigitaljimnope18:55
imdigitaljimits not there18:56
flwang1upstream?18:56
flwang1you mean the 'upstream'?18:56
flwang1yep, it's not there18:56
imdigitaljimhttps://github.com/coredns/coredns/blob/master/plugin/kubernetes/README.md18:56
flwang1cool, let me try18:57
flwang1strigazi: can you help try on your side?18:58
flwang1wait19:01
flwang1seems the option 'upstream' is used to resolve domain name out of pod19:02
flwang1but the name 'kubernetes' is a internal name, no?19:02
flwang1imdigitaljim: ^19:02
colin-yes19:02
strigazii added 'upstream' no change19:03
colin-the line two above 'upstream' supports resolution of that record19:03
colin-the one begining with 'kubernetes'19:03
colin-https://coredns.io/plugins/kubernetes/19:06
flwang1and based on the command result, the pod can reach the coreDNS pod, but coreDNS pod failed to parse 'kubernetes', so probably something wrong between coredns and k8s?19:06
strigazihmm, heapster is resolved with nslookup19:07
strigazibut kubernetes no19:07
colin-the NXDOMAIN response in your paste certainly suports an inability to look that record up19:07
colin-can you share the output of your coredns configmap? kubectl -n kube-system get configmap coredns -o yaml19:08
colin-oops did not mean to paste the command, was on my clipboard19:08
flwang1http://paste.openstack.org/show/731517/19:09
flwang1strigazi: does that mean nslookup should also work for other user customized service?19:10
colin-is the dnsPolicy set on your busyboxy pod?19:10
strigazidnsPolicy?19:10
colin-https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-s-dns-policy19:11
flwang1we didn't do anything about the pod19:11
flwang1just the original image to create the pod19:11
colin-so default probably19:12
colin-The Pod inherits the name resolution configuration from the node19:12
colin-if the node isn't consulting the coredns service by default then the busyboxy pod may not for the same reason19:12
imdigitaljimdnsPolicy: ClusterFirstWithHostNet i think he might be referring19:13
imdigitaljimwanna try that?19:13
*** ykarel__ has joined #openstack-containers19:13
flwang1im trying19:13
*** ykarel__ has quit IRC19:14
*** ykarel__ has joined #openstack-containers19:14
*** ykarel_ has quit IRC19:15
colin-or maybe ClusterFirst since you aren't using host networking on that busybox pod19:15
flwang1colin-: good point19:16
*** ykarel_ has joined #openstack-containers19:17
flwang1http://paste.openstack.org/show/731519/19:19
flwang1no luck19:19
flwang1strigazi: ?19:19
*** ykarel__ has quit IRC19:20
colin-10.254.0.10 is the ClusterIP for your 'kube-dns' svc right?19:20
flwang1yes19:20
strigaziyes19:20
flwang1http://paste.openstack.org/show/731520/19:21
*** ykarel__ has joined #openstack-containers19:21
flwang1i have to go to office in mins, will be back soon. only slept 3 hours, very sleepy, need some coffee now19:22
*** ykarel_ has quit IRC19:23
strigaziit seems that it worked now19:24
strigaziI'll do one more test19:25
*** ykarel__ has quit IRC19:26
flwang1how?19:29
strigazipods in kube-system can not reach the kubernetes service19:29
strigazionly in the default ns19:29
strigaziI'll validate the config of coredns once more19:30
flwang1is it because some network policy of calico?19:30
strigaziprobably19:31
strigazibut i don't have experience with it19:31
flwang1but the busybox is in default ns19:31
flwang1i still can't get the point19:31
flwang1i have to run now19:31
flwang1will be back in 30 mins19:32
strigaziok19:32
*** flwang1 has quit IRC19:36
*** mattgo has joined #openstack-containers19:37
strigaziexport KUBECONFIG=/opt/stack/clusters/kube/config19:40
*** serlex has quit IRC19:40
strigazii'm getting crazy here, in a centos pod i can nslookup kubernetes but in a busybox?19:41
strigazii also pasted my clipboard? cool19:42
*** slagle has quit IRC19:43
strigaziimdigitaljim: colin- can you have a look into this: http://paste.openstack.org/raw/731522/19:52
strigaziIt really doesn't make sense19:53
strigazibusy-debug is busybox19:53
strigazidebug-1 is custom alpine with nslookup installed19:54
strigazicentos-1 is gitlab-registry.cern.ch/linuxsupport/cc7-base where i installed bind-utils19:54
strigazior this one ^^ docker pull cern/cc7-base19:55
*** slagle has joined #openstack-containers20:19
*** flwang has joined #openstack-containers20:24
flwangstrigazi: still around?20:24
flwangimdigitaljim: any message i missed?20:24
strigaziflwang: no changes wfm http://paste.openstack.org/raw/731523/20:31
strigazibusybox didn't work20:31
flwangbut centos works?20:32
strigazicentos and alpine in the example in the paste20:32
imdigitaljimyeah ive just been diving around, i havent seen anything that stands out20:32
flwanginteresting, that's enough i think20:32
imdigitaljimstrigazi: ^ that sounds promising20:32
flwangso only the busybox doesn't work, right?20:32
strigaziyes20:33
flwangfor me, i think centos and alpine are enough for us to continue the release20:33
strigazimaybe there is bug in busybox?20:33
flwangprobably20:33
strigaziwith flannel in me devstack i had the same problem20:34
strigazis/me/my20:34
flwangsame problem, means busybox aslo not working?20:34
strigazitry with quay.io/strigazi/alpine-nslookup20:34
flwangthen it should be a problem of busybox20:34
strigaziFROM alpine:edge RUN apk add bind-tools20:35
strigaziflwang: can you verify?20:36
flwangi'm verifying20:36
strigazithat works for you20:36
flwangshit, do you guys know how to clean the dns server cache?20:38
*** devananda has quit IRC20:38
*** slagle has quit IRC20:39
flwangi'm getting Error from server: error dialing backend: dial tcp: lookup test-final-6-deipcjvrcnnl-minion-0 on <my dns server>:53: no such host20:40
strigaziwhich dns is that?20:43
flwangit's our preprod dns server20:43
flwangi mean <my dns server>20:43
flwangi'm running the command on master20:44
flwangi don't know why it's trying to talk to our local dns server20:44
strigaziit shouldn't20:45
flwangyep, i know20:52
strigazithis must be a config option in pod20:52
flwangstrigazi: imdigitaljim: colin-:  THANK YOU for all you guys help20:53
flwangi really really appreciate it20:53
flwangi will buy you guys a beer at Berlin20:53
strigaziflwang: anytime, let us know how it goes :)20:53
strigaziI'm going offline, see you later20:54
flwangi will send email to update the status20:54
strigazithanks20:54
flwanggood night20:54
strigazihave a nice day :)20:54
imdigitaljimnp see ya!20:54
imdigitaljim:D20:54
*** itlinux has quit IRC21:14
*** itlinux has joined #openstack-containers21:17
*** devananda has joined #openstack-containers21:25
*** itlinux has quit IRC22:05
*** itlinux has joined #openstack-containers22:22
*** mattgo has quit IRC22:23
*** itlinux has quit IRC22:25
*** rcernin has joined #openstack-containers22:28
colin-hope it's working now flwang :)22:49
flwangcolin-: it's working now22:50
flwangjust something wrong with the busybox images22:50
flwangcolin-: thank you for all the help22:50
colin-of course, any time22:56
colin-glad to hear it22:56
*** hongbin has quit IRC22:57
flwangcolin-: cheers23:04
*** slagle has joined #openstack-containers23:35
« Wednesday, 2018-10-03 Index Friday, 2018-10-05 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!