Tuesday, 2017-03-21

kfox1111heh. helps if ou write it to an actual file....00:00
kfox1111and look in the right place...00:01
openstackgerritKevin Fox proposed openstack/kolla-kubernetes master: WIP: Fernet Token Support  https://review.openstack.org/44627400:02
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla-kubernetes master: WIP: Playbook to add configmaps and secrets  https://review.openstack.org/44773300:02
inc0kfox1111: told you it's going to be quick;)00:02
sdakevhosakot why no workflow on https://review.openstack.org/#/c/447217/00:03
*** eaguilar has quit IRC00:03
vhosakotsdake: sorry forgot, done!00:03
sdakevhosakot roger thanks :)00:03
vhosakotsdake: np00:03
vhosakot:)00:03
kfox1111inc0: where's the proof of it working? ;)00:04
inc0well I did create secret in k8s;)00:04
* sdake lols00:05
kfox1111inc0: I'm not you. how do I review it?  probably shoudl tweak the gate scripts to test it.00:05
inc0kfox1111: WIP is in title00:05
kfox1111ok. :)00:05
inc0I just want to get your opinion on approach00:05
kfox1111it just sounded like you throught it was done.00:05
kfox1111k. sec...00:05
sdakeinc0 integrating in the gate will be miles harder then doing the implemetation00:06
inc0sdake: nah00:06
sdakeinc0 zhubingbing was planning to rework that cherrypick to integrate with the gate by removing the awful symlinks in teh gate00:06
sdakewhich are a source of great confusion00:06
sdakei.e. two birds one stone00:06
kfox1111it doesn't get rid of gencofnig, so won't be too hard I think.00:06
inc0nah, it assumes genconfig was run and rendered files are available somewhere00:07
inc0it really just templates configmaps00:07
kfox1111I think there may be devils in the details. we'll see what falls out of the gate though.00:07
kfox1111so... how much extra time is this going to add to the gates?00:08
kfox1111its like a couple minutes just for genconfig right now.00:08
inc0kfox1111: locally it runs in less than a second00:09
kfox1111looking through the code, I guess its ok looking to me. it will just shift code from the python command to the ansible one.00:09
kfox1111ah. cause its ownly handilng the one file.00:09
inc0yeah....much much simpler ansible one00:09
inc0and we can examine output at the end00:10
kfox1111I don't expect that to wokr. most configmaps are multiple files.00:10
kfox1111might have to tweak that a bit.00:10
inc0sure, I'll figure it out00:10
inc0not a hard one to solve too00:10
kfox1111yeah.00:10
inc0ok I'm off, tomorrow I'll finish it up and add it to gate00:11
kfox1111so... most of this could be replaced by just a shell script with00:11
kfox1111kubectl create configmap ....00:11
inc0well, rendering template is hard to shell-up00:12
kfox1111that may be more understandable.00:12
inc0and also kubectl -f /path/to/dir will check all files in dir with .yaml at the end and add all of them00:12
inc0so it's one command00:12
inc0kubectl create -f /tmp/kolla-kubernetes/configmaps00:13
inc0will add all of them00:13
kfox1111so is kubectl create configmap --namespace kolla foo-configmap --from-file=/etc/kolla/foo ??00:15
inc0yeah00:15
inc0no00:15
inc0sorry00:15
inc0kubectl create --namespace kolla -f /etc/kolla/foo00:15
kfox1111thats not valid.00:16
inc0you don't create each configmap00:16
inc0why?00:16
kfox1111kubectl create configmap --namespace kolla foo-configmap  --from-file=/etc/kolla/foo00:16
kfox1111is though.00:16
inc0no, no need00:16
kfox1111create takes in a k8s yaml file.00:16
inc0if you create yaml correctly00:16
inc0this playbook renders yaml file00:16
inc0for k8s00:16
kfox1111create configmap generates those objects automatically.00:16
inc0that's what's its form00:16
inc0for00:16
vhosakotinc0: yeah, you can add the configmap in YAML spec and add use "kubectl create -f <yaml spec with configmap>"00:17
kfox111190% of what kollakube res create configmap does is kubectl create configmap ...00:17
inc0and all of it can be replaces with this simple play...00:17
kfox1111the other 10% is actually rendering templates of its own.00:17
kfox1111really not sure what ansible's buying us, other hten more ansible.00:18
inc0kollakube is buggy00:18
kfox1111inc0: how so?00:18
inc0kollakube is not super convinient00:18
vhosakotkfox1111: saw your message in the sig-ui slack channel about dashboard.  are you able to install the dashboard with service account successfully on kubeadm?  I'm seeing errors.00:18
inc0when I run kollakube today it started to look for neutron in wrong places00:19
kfox1111vhosakot: I've only tried 1.6 in a static pod configuration. it did work there.00:19
inc0so I looked at code and cried a little00:19
kfox1111inc0: then why is it working for everyone else?00:19
inc0what this gives us is that you'll be able to examine/edit yamls afterwards00:19
kfox1111yeah, the code isn't very pretty any more. :/00:19
vhosakotkfox1111: ah, static pod config.. cool.. I'll try service account and will ask you help if needed tomorrow..00:19
kfox1111vhosakot: cool.00:20
kfox1111vhosakot: btw,00:20
inc0which is additional benefit00:20
inc0kfox1111: trust me on that one plz, we don't want to have custom cli any more00:20
kfox1111vhosakot: https://github.com/kubernetes/charts/pull/80800:20
inc0rewriting it to proper state will be hard by now00:20
kfox1111inc0: its on the way out. why spend any more time on it? what does it actually buy us though?00:21
inc0this ansible gets us there faster and in much more reliable way00:21
kfox1111users can edit /etc/kolla/* today.00:21
inc0well, yamls are better for that imho00:21
kfox111199% of the problems in that code path today are genconfig induced breakages. :/00:21
vhosakotkfox1111: great, thanks for the link.  my "helm init" is broken too, after I fix it, I'll install the dashboard chart and post comments in the review.00:21
kfox1111vhosakot: I'm guessing then its a sdn issue.00:22
inc0this won't help with genconfig00:22
kfox1111vhosakot: as helm init requires sdn.00:22
kfox1111inc0: exactly my point.00:22
inc0it's meant to solve different problem00:22
inc0namely kollakube00:22
kfox1111lets fix the genconfig problem, not the 1% little tiny part in the middle. :/00:22
inc0which imho is bad00:22
inc0but fixing genconfig will be MUCH longer00:22
kfox1111which fixes itself when the genconfig problem is fixed.00:23
inc0I mean I did that in an hour.00:23
kfox1111inc0: yeah, but its far far from done.00:23
inc0not that far00:23
vhosakotkfox1111: ah, I'm using canal.. is canal just CNI or CNI+SDN?  what are you using for pod networking? canal?00:23
inc0anyway, I'll prove it to you tomorrow00:23
kfox1111add it to the gate. we'll see. :)00:23
kfox1111ok. :)00:23
inc0for now, I'm going home00:23
inc0have a good one guys00:23
kfox1111vhosakot: canal. yeah. thats what I was meaning.00:23
kfox1111inc0: ok. have a good evening. :)00:23
*** masber has joined #openstack-kolla00:24
vhosakotkfox1111: yeah, I'm using canal and tested few apps, so cluster is in working condition. are you able to run "helm init" successfully on kubeadm.  sdake and I debugged it last week, and looks like the tiller was in CrashLoopbackook,  I'll debug it tomorrow, was pulled into other things today.00:25
kfox1111vhosakot: yup.00:25
kfox1111the gate's running it too.00:25
vhosakotkfox1111: cool, thanks for the info.00:25
kfox1111kubeadm + canal + helm.00:25
vhosakotsweet!00:25
vhosakotyeah, I saw the gate logs today00:26
*** jrobinson has joined #openstack-kolla00:28
openstackgerritKevin Fox proposed openstack/kolla-kubernetes master: WIP: Fernet Token Support  https://review.openstack.org/44627400:30
*** vhosakot_ has joined #openstack-kolla00:31
*** Margin has joined #openstack-kolla00:33
*** vhosakot has quit IRC00:33
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP Adding virtualbmc to the gate  https://review.openstack.org/44155000:34
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP Adding virtualbmc to the gate  https://review.openstack.org/44155000:38
*** Pavo has joined #openstack-kolla00:40
sbezverkkfox1111: by some reason gate does not like echo "0" /proc/blah commands00:43
sbezverkis there a different way? maybe using sysctl or something?00:44
sdakesbezverk did you try sudo?00:44
sdakevhosakot_ note we ran helm 2.2.2 yesterday - not sure what version kfox1111 is running00:45
*** Pavo has quit IRC00:45
*** duonghq has joined #openstack-kolla00:45
sbezverksdake: I have sudo in front of this command00:45
duonghqmorning guys00:45
sdakesbezverk echo may not be in the sudoers file00:45
sdakesbezverk infra tunes the images so that they are secure from damaging other infra vms00:46
sdakesbezverk an echo to /proc could potentially muck with the infrastructure00:46
sdakesbezverk i'd recommend asking in openstack-infra00:46
sdakesup duonghq00:47
vhosakot_sdake: right, I saw the issue with 2.1 as well. I see you've pinned k8s version to 1.5.2 and helm version to 2.2.2 in the docs.. cool00:48
sdakekfox1111 FWIW kollakube does not work for bare metal deployments with the symlink workflow00:48
sdakekfox1111 although I agree i struggle with the purpose of  ansible in this code path as well00:49
kfox1111sdake: why not?00:49
*** Pavo has joined #openstack-kolla00:49
kfox1111sbezverk: oh.... tahts a root level command.00:50
*** adrian_otto has quit IRC00:50
*** Margin has quit IRC00:50
vhosakot_sdake: thanks for pinning versions in https://review.openstack.org/#/c/447164/1/doc/source/development-environment.rst.  less variables in docs now, cool.00:50
duonghqhow do you setup vip in multinode setup?00:50
*** sayantani01 has joined #openstack-kolla00:50
kfox1111try "echo 0 | sudo dd of=/proc...."00:50
duonghqwith or without haproxy enable?00:50
vhosakot_kfox1111: are you using helm 2.2.2?00:50
kfox1111vhosakot_: yeah.00:50
vhosakot_kfox1111: cool00:50
kfox1111oh. gotoa head out. talk to you all later.00:50
vhosakot_o/00:51
sdakekfox1111 see this review: https://review.openstack.org/#/c/439740/00:51
*** Margin has joined #openstack-kolla00:51
sdakekfox1111 the symlinks cause kolla-kube to break wildly when pip install is used00:51
sdakekfox1111 look at the gate results00:51
sdakekfox1111 yet without that patch mariadb configmap cannot be created - this is what triggered this whole discussion00:51
kfox1111sdake: the gates doing symlinks though.00:52
kfox1111and it pip installs it I think?00:52
sdakeinc0 is reliving the experience I had 2 weeks ago00:52
sdakekfox1111 the gate uses symlinks00:52
sdakeI didn't use symlinks00:52
masberhow can i find the ports, the kolla containers are listening?00:52
sdakeoeprators wont use symlinks00:52
kfox1111oh.00:52
kfox1111well, use symlinks :)00:52
sdakewrong answer00:52
kfox1111I'm an op, and I did.00:52
* sdake groans loudly00:52
kfox1111right answer is to get out of genconfig. :/00:52
sdakeagreed00:52
sdakeshort term answer is to get rid of symlinks in the gate00:52
kfox1111lets stop tilting at the other windmill.00:52
kfox1111or use a container! :/00:53
sdakesymlinks are not a shiny object00:53
kfox1111put all the things in a container, symlink away and who cares. :/00:53
sdakei agree that is a medium term objective00:53
kfox1111it buys us time to fix it right, and makes it easier for folks to pull a dev/prod env.00:53
sdakegetting there is not easy00:53
kfox1111the dev env already does it.00:53
kfox1111the works largely been done.00:54
masberI am experiencing this isssue --> https://bugs.launchpad.net/kolla-ansible/+bug/1626339 and http://paste.openstack.org/show/603543/00:54
openstackLaunchpad bug 1626339 in kolla-ansible "Kolla mongodb error: can't find self in the replset config" [High,In progress] - Assigned to Jeffrey Zhang (jeffrey4l)00:54
sdakepeople dislike the dev env becasue - its based on vagrant00:54
*** Pavo has quit IRC00:54
vhosakot_^^^^00:54
kfox1111agreed. but the container != vagrent.00:54
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP Adding virtualbmc to the gate  https://review.openstack.org/44155000:54
sdakekfox1111 we could chase that shiny object for months and still not have an onramp for new devs00:54
vhosakot_I saw vagrant issues nested-virtualizaion issues for 2 weeks, and now use bare-metal VMs.00:55
sdakekfox1111 ^^00:55
vhosakot_sdake: the dev env minus vagrant is awesome.00:55
kfox1111sdake: or we could rewrite kollakube and spend a bunch of time avoiding a simple symlink. :/00:55
kfox1111bigger fish.00:55
sdakei dont think kollakube needs to be rewrritten00:56
masberI was wondering if someone here could give me a workaround, I can't put storage_interface and api_interface on the same interface because my users don't have access to the storage network so I need to separate them if I want them to access Horizon00:56
sdakeit is a 2 line patch00:56
sdakehttps://review.openstack.org/#/c/439740/00:56
sdakesymlinks on the other hand need to be fixed00:56
kfox1111sdake: seeing a bunch of "simple" patches today that break everything. :)00:56
sdakebreaks the gates.00:57
sdakewhy is that?00:57
kfox1111though if the gate's fixed, I'm not against it.00:57
sdakethere ya go00:57
sdakezhubingbing has agreed to fix the symlink issue this week00:57
kfox1111sdake: https://review.openstack.org/#/c/439740/1 a lot of red.00:57
sdakeso we can have a proper deploy guide00:57
sdakekfox1111 a lot of red all around symlinks00:58
sdakeremove the symlinks - use pip install - golden00:58
sdakekfox1111 that patch is actually correct - kolla-kube is defective, the patch needs expansion to fix the gate00:58
kfox1111fix all the stuff it breaks, and I'll +2 it. :)00:58
sdakekfox1111 zhubingbing will do so :)00:59
kfox1111ok.00:59
sdakekfox1111 before ou head ou tfor  the day you leeft a comment on line 4201:01
*** tonanhngo has quit IRC01:01
sdakeof m=the doc guide01:01
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP Adding virtualbmc to the gate  https://review.openstack.org/44155001:01
sdakehttps://review.openstack.org/#/c/447356/5/doc/source/deployment-guide.rst01:02
sdakekfox1111 how precisely do I prune the example01:02
openstackgerritKevin Fox proposed openstack/kolla-kubernetes master: WIP: Fernet Token Support  https://review.openstack.org/44627401:03
kfox1111sdake: remove everythign thats a default.01:03
sdakekfox1111 the defaults are in all_values.yaml?01:03
kfox1111see line 40.01:03
kfox1111it should spit out all the defaults.01:04
kfox1111gota head out. l8r.01:04
sbezverkkfox1111: have a good one01:05
*** krtaylor has joined #openstack-kolla01:06
openstackgerritpangliye proposed openstack/kolla-ansible master: Delete the copy operation of telegraf plugin  https://review.openstack.org/44774201:09
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP Adding virtualbmc to the gate  https://review.openstack.org/44155001:10
duonghqsdake, how do you setup vip for multinode setup?-01:14
*** iceyao has joined #openstack-kolla01:16
duonghqsbezverk, how do you setup vip for multinode setup?01:16
sdakeduonghq do you mean for kolla-kubernetes?01:18
duonghqsdake, yes01:18
sdakeduonghq no clue - if you find out do share :)01:18
sdakeduonghq i'd like to just get 1 master 1 node working correctly01:19
duonghqso, what IP do you set for VIP?01:19
duonghqwhich node?01:19
sdakeduonghq can you review this: https://review.openstack.org/#/c/447356/01:19
duonghqsdake, roger01:20
sdakeduonghq how are you setting vip in the first place?01:20
sdakeduonghq maybe that review contains the answer :)01:20
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP Adding virtualbmc to the gate  https://review.openstack.org/44155001:21
sbezverkduonghq: yes01:21
*** tonanhngo has joined #openstack-kolla01:22
sbezverkduonghq: sorry I meant to say I was using haproxy01:22
duonghqsbezverk, nice, so I can set vip to any IP in the subnet?01:23
sbezverkwhich was in front of several api server01:23
sbezverkservers01:23
*** tonanhngo has quit IRC01:23
*** eaguilar has joined #openstack-kolla01:24
sbezverkduonghq: as long as you can stich vip with one of kube-api ip addresses, you should be ok01:24
duonghqsbezverk, understood01:24
duonghqit cannot be arbitrary IP?01:24
sbezverkduonghq: I think it should be from the same subnet01:26
sbezverknever tried fancy stuff here01:26
sbezverkenough problems already01:26
duonghqsbezverk, sure, but if in same subnet, as I tried, seem that it does not show up01:26
duonghq*if just in same subnet01:27
duonghqas in this paste: http://paste.openstack.org/show/603545/01:27
*** goldyfruit has joined #openstack-kolla01:27
sbezverkduonghq: hmm, I will have to dig up config for haproxy I used with multinode01:27
duonghqI don't know why keystone-admin doesn't got external ip01:28
sbezverkduonghq: what is this ip 192.168.122.248   ?01:29
duonghqthis is IP kube master01:29
sbezverkduonghq: you have to explicetely say to use external for admin01:29
sbezverkkubectl get svc -n kolla | grep key01:30
sbezverkkeystone-admin       10.57.188.241   192.168.80.232               35357/TCP   11h01:30
sbezverkkeystone-internal    10.57.168.139   192.168.80.232               5000/TCP    11h01:30
sbezverkkeystone-public      10.57.186.117   192.168.80.232               5000/TCP    11h01:30
sbezverkduonghq: I think you are missing a couple of config settings01:30
*** zhubingbing_ has joined #openstack-kolla01:30
duonghqsbezverk, I have brief meeting now, sorry, brb01:30
*** japestinho has joined #openstack-kolla01:30
duonghqya, will find it01:30
*** iceyao has quit IRC01:33
*** iceyao has joined #openstack-kolla01:38
*** rhallisey has quit IRC01:39
*** Pavo has joined #openstack-kolla01:40
*** iceyao has quit IRC01:43
*** Pavo has quit IRC01:43
*** tonanhngo has joined #openstack-kolla01:45
duonghqI'm back01:45
*** cuongnv has joined #openstack-kolla01:45
*** tonanhngo has quit IRC01:46
*** saneax-_-|AFK is now known as saneax01:49
openstackgerritKevin Fox proposed openstack/kolla-kubernetes master: WIP: Fernet Token Support  https://review.openstack.org/44627401:51
*** crushil has joined #openstack-kolla01:51
duonghqsbezverk, which tool do you use for setup k!S cluster01:51
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP Adding virtualbmc to the gate  https://review.openstack.org/44155001:53
duonghqseem that dns in kargo failed me01:55
kfox1111sbezverk: your about to hit 100 revisions. :)01:55
*** zhurong has joined #openstack-kolla01:57
openstackgerritpangliye proposed openstack/kolla-ansible master: Delete the copy operation of telegraf plugin  https://review.openstack.org/44775201:58
sbezverkkfox1111: yeah, lots of hick ups along the way01:58
*** pramodrj07 has joined #openstack-kolla01:59
sbezverkduonghq: I do everything maunally using kubeadm01:59
kfox1111sbezverk: :) its a hard bit of code to get right. but its coming out awesome. great job on it dude.02:00
sbezverkkfox1111: :) thanks.. btw it turned out, gate vm was mnissing br_filter modules02:00
sbezverkthat was why echo 0 was failing02:00
*** pramodrj07 has quit IRC02:01
kfox1111thats ironic. :)02:02
*** MasterOfBugs has quit IRC02:02
duonghqsbezverk, how about network plugin? which one do you use?02:03
*** rwellum has joined #openstack-kolla02:05
*** adrian_otto has joined #openstack-kolla02:15
openstackgerritKevin Fox proposed openstack/kolla-kubernetes master: WIP: Fernet Token Support  https://review.openstack.org/44627402:17
*** ipsecguy_ has quit IRC02:22
*** vhosakot_ has quit IRC02:22
*** ipsecguy has joined #openstack-kolla02:22
*** vhosakot has joined #openstack-kolla02:23
*** tonanhngo has joined #openstack-kolla02:27
*** vhosakot has quit IRC02:27
*** tonanhngo has quit IRC02:28
*** saneax is now known as saneax-_-|AFK02:29
*** unicell1 has quit IRC02:31
*** vhosakot has joined #openstack-kolla02:33
*** goldyfruit has quit IRC02:35
*** saneax-_-|AFK is now known as saneax02:47
*** masber has quit IRC02:54
*** zhurong has quit IRC03:05
*** adrian_otto has quit IRC03:06
*** tonanhngo has joined #openstack-kolla03:09
*** tonanhngo has quit IRC03:11
*** masber has joined #openstack-kolla03:15
openstackgerritSayantani Goswami proposed openstack/kolla-ansible master: [WIP] Added playbooks for snap deployment  https://review.openstack.org/44616503:19
*** zhurong has joined #openstack-kolla03:22
*** eaguilar has quit IRC03:23
*** dave-mccowan has quit IRC03:29
*** adrian_otto has joined #openstack-kolla03:32
*** saneax is now known as saneax-_-|AFK03:40
*** sayantan_ has joined #openstack-kolla03:44
*** sayantani01 has quit IRC03:47
*** crushil has quit IRC03:50
sdakesup peeps03:53
sdakesbezverk surprised you can modprobe in the gate03:53
masberhi03:59
masberis the kuryr image ready?03:59
masberdeploy is saying that it can't find it03:59
*** tovin07_ has joined #openstack-kolla04:00
*** adrian_otto has quit IRC04:06
*** adrian_otto has joined #openstack-kolla04:07
duonghqsdake, which network plugin are you using for k8s cluster?04:08
*** zhurong has quit IRC04:12
*** tonanhngo has joined #openstack-kolla04:12
*** tonanhngo has quit IRC04:13
masberI amalso getting this error when deploying ironic --> http://paste.openstack.org/show/603557/04:14
sdakeduonghq no clue04:24
sdakemasber it was tested by someone in the community although I don't know exactly how :)04:25
duonghqsdake, can you ping to any cluster if or services from within one pod?04:25
sdakeduonghq i think canal but can't confirm04:25
sdakeduonghq i dont have access to my env atm as it is torn down attempting to test this document :)04:25
masbersdake, no worries, I am just wondering if this was working for someone else or was a mistake I did04:26
duonghqroger04:26
masberby the way I am using kolla-ansible04:26
sdakemasber the answer to your quesiton is yes it was working for someone else - the assumption that it is a mistake you did is possibly wrong :)04:26
sdakemasber in other words, it may be broken in master of kolla-ansible as the gating of kolla-ansible specifically is pretty weak04:26
masberI see, thanks for the clarification04:27
masberwhat about kurys?04:27
sdakezhubingbing_ you alive?04:27
masberI can't even pull it04:27
masber*kuryr04:27
zhubingbing_sdake i am here04:27
sdakemasber which version of kolla-ansible are you using?04:27
masber4.0.004:28
sdakemasber distro choice?04:28
masbercentos binary04:28
sdakekuryr is not available in centos binary04:28
sdaketry centos source04:28
masberok!04:29
*** adrian_otto has quit IRC04:29
masberalso I am having this issue https://bugs.launchpad.net/kolla-ansible/+bug/162633904:29
openstackLaunchpad bug 1626339 in kolla-ansible "Kolla mongodb error: can't find self in the replset config" [High,In progress] - Assigned to Jeffrey Zhang (jeffrey4l)04:29
masberseparating api_interface and storage_interface will make mondodb to break while deployment04:30
sdakezhubingbing_ there is renewed vigor in resolving the gating with https://review.openstack.org/#/c/439740/04:30
masberI tried changing the ip on the ansible command so both points to the same IP but still didn't work04:30
sdakemasber no clue on that one :)04:31
masberis there a way I can get in contact with Jeffrey Zhang?04:31
masberI would like to assist him on testing this issue04:31
masberand see if we can get it running soon04:31
sdakemasber what i can tell you is the bug is in progress and will always be resolved by Jeffrey4l_ when he takes it on :)04:31
*** skramaja has joined #openstack-kolla04:32
masbersure04:32
zhubingbing_ok04:32
sdakebmace you alive?04:32
masberI will try to talk to him and see if I can help him testing or something04:33
masberthanks sdake!04:33
sdakemasber happy to help04:33
zhubingbing_sdake i fix it04:33
*** manheim has joined #openstack-kolla04:33
openstackgerritSurya Prakash (spsurya) proposed openstack/kolla-kubernetes master: Deleting the deprecated option  https://review.openstack.org/44778904:35
duonghqsdake, with the guide, I still got this error: Failed to contact the endpoint at http://keystone-internal:5000/v3 for discovery. Fallback to using that endpoint as the base url.04:36
duonghqNot Found (HTTP 404)04:36
duonghq04:36
Jeffrey4l_mnaser, i am around ;)04:36
sdakeduonghq did you use kargo to deploy?04:36
duonghqyup, guess that it messes up with dns, cannot ping to cluster ip from inside of any pods04:37
sdakeduonghq gotcha - can you do this kubectl get pods | grep dns04:37
*** saneax-_-|AFK is now known as saneax04:37
sdakekubectl get pods -n kolla | grep dns04:37
*** manheim has quit IRC04:37
Jeffrey4l_mnaser, here it the fix https://review.openstack.org/#/c/442999/04:38
sdakeduonghq its good that everything came up :)04:38
Jeffrey4l_no idea how infra-bot do not add it into bp.04:38
duonghqsdake, http://paste.openstack.org/show/603559/04:38
Jeffrey4l_lp*04:38
sdakeJeffrey4l_ sometimes launchpad rate limits and launchpad misses changes04:38
Jeffrey4l_got. thanks.04:39
sdakeJeffrey4l_ its not an infra problem its a launchpad problem04:39
sdakeduonghq moment:04:39
masbersdake, would you recommend using kolla 3.0.2 instead of 4.0.0?04:41
sdakeduonghq see this: https://github.com/openstack/kolla-kubernetes/blob/master/tools/setup-resolv-conf.sh#L604:41
sdakemasber i'd recommend 4.0.004:41
sdakemasber although 3.0.2 is more stable - depends if your eval or deploying04:42
masberapparently centos-source-kolla-toolbox:4.0.0 is missing from the centos source repository04:42
masberI am deploying04:42
sdakeJeffrey4l_ did you get a chance to push the 4.0.0 images?04:42
masberso I need stability, happy to upgrade once 4.0.0 is stable04:42
Jeffrey4l_sdake, i have pushed all 4.0.0 images.04:42
duonghqsdake, it can get the ip but not ping to this ip04:43
Jeffrey4l_except centos+source, iirc you have pushed this, right?04:43
sdakeJeffrey4l_ it is entirely possible i did it wrong - my network was flakey04:43
sdakelet me check dockerhub in a minute04:43
sdakemasber ^^04:43
masbermmm I am not sure if I am using docker hub04:44
sdakemasber did you build your own images?04:44
sdakeduonghq see line 6 of that github resolv.conf?04:44
masbersdake, nop egonzalez gave me another repo04:44
duonghqyup04:44
sdakethat needs to be kubedns04:44
sdakekargo uses kubedns - kubeadm uses kube-dns04:44
duonghqI already change kude-dns to kubedns04:45
sdakewhy have one special snowflake when you can have a blizzard04:45
sdakemasber please use dockerhub04:45
sdakemasber not sure what egonzales gave you04:46
sdakeduonghq did you run setup-resolv-conf.sh?04:46
duonghqyup04:46
sdakewhen did you run it?04:46
sdakeafter a clean env setup?04:47
duonghqbefore 1st helm install04:47
duonghqya04:47
duonghqshould I rerun all of this again?04:47
duonghqcleaning helm04:47
duonghqand reinstall all04:47
sdakemoment before you proceed04:47
masberlet me double check04:47
*** unicell has joined #openstack-kolla04:48
sdakemasber paste your globals.yml04:48
masberhttp://paste.openstack.org/show/603560/04:49
sdakemasber : docker_registry: "129.94.72.62:4000"04:50
sdakethat looks like a local registry of some sort04:50
sdakemasber i suspect but can't confirm you didn't push centos source images there04:51
sdakemasber this would cause docker pulls to fail as the images are not in the remote registry04:51
masberyes04:52
masberneed to fid out which is the remote registry04:52
sdakemasber is that some internal registry that egonzalez setup?04:53
masberI followed his isntructions04:53
sdakemasber yup - where is that registry stored?04:53
masberhttp://egonzalez.org/04:53
sdakeimo file a bug - BLOG POST :)04:54
masberwget https://tarballs.openstack.org/kolla/images/centos-binary-registry-ocata.tar.gz04:54
masberlooks like this registry is part of kolla04:55
sdakedocker_registry: "172.28.128.3:4000"04:55
masber*container docker registry04:55
sdakethat needs to be customized to your local environment04:55
sdakeConfigure kolla globals.yml configuration file with the following content.04:55
sdakeChange values when necessary (IP addresses, interface names).04:55
sdakeThis is a sample minimal configuration.04:55
masberyes, that IP is the host I am running that container04:55
masberI changed taht04:56
sdake# wget https://tarballs.openstack.org/kolla/images/centos-binary-registry-ocata.tar.gz04:56
sdakethat will download centos-binary04:56
duonghqsdake, I cannot even ping the nameserver in resolev.conf04:56
sdakemasber you need to kill yoru registry and get centos-source-registry-ocata.tar.gz04:56
masberhttp://paste.openstack.org/show/603561/04:57
masberok ok04:57
masberbasically my registry does not have the centos source that is why deployment is failing04:58
masberok04:58
masberso I need to understand, what policy do you follow to push containers to binary or source?04:59
masberare source like for testing?04:59
*** zhurong has joined #openstack-kolla05:00
sdakemasber i have other things to do at tthis moment - i can help in the morning perhaps - or ping the emea crew when they arrive :)05:00
masbersure, np05:01
sdakeduonghq can you show me your resolv.conf05:01
sdakeduonghq in the containers05:02
duonghqsdake, here http://paste.openstack.org/show/603562/05:02
sdakeduonghq give this command a spin outside the container:05:05
sdakekubectl get svc --namespace=kube-system -l k8s-app=kubedns -o jsonpath='{.items[*].spec.clusterIP}'05:06
duonghqsdake,  10.233.0.305:07
duonghqsdake, kubectc get svc output: http://paste.openstack.org/show/603563/05:08
sdakeduonghq looks like your dns is setup properl yas keystone-admin is resolving05:10
sdakei think you have a networking problem05:10
duonghqsdake, I can ping pod of other services directly05:11
duonghqbut not through cluster ip05:11
sdakedefine ping pod of other services directly?05:12
duonghqi.e. I can ping mariadb-0 pod, but not mariadb service05:12
duonghqinside keystone pod05:12
sdakewhat ip is mariadb service?05:13
duonghqmariadb             10.233.25.205   <none>            3306/TCP    22h05:13
duonghqpod: 10.233.97.15205:13
sdakecan you paste kubectl get pod -n kolla | grep mariadb05:14
duonghqmariadb-0                    1/1       Running   0          22h05:14
sdakekubectl describe pod mariadb-0 -n kolla?05:16
duonghqhttp://paste.openstack.org/show/603564/05:16
sdakeIP:10.233.97.15205:22
sdakeyou can ping 10.233.97.152?05:23
sdakewhich IP can't you ping?05:23
duonghqsdake, the IP of  mariadb             10.233.25.205   <none>            3306/TCP    22h05:25
duonghq10.233.25.20505:25
sdakeduonghq not sure why that is05:26
duonghqya05:26
sdakeduonghq gotta hit the rack05:26
sdakeduonghq try sbezverk in the morning - he can probably get you going05:27
sdakeduonghq if you sort it out - i'd like to know05:27
duonghqsdake, thanks, my keystone cannot even connect to its internal endpoint05:27
duonghqnight sdake05:28
sdakenight05:28
*** satyar has joined #openstack-kolla05:29
*** lazyPower has quit IRC05:32
*** lazyPower has joined #openstack-kolla05:32
*** lazyPower has quit IRC05:33
*** lazyPower has joined #openstack-kolla05:33
*** zhurong has quit IRC05:40
*** tonanhngo has joined #openstack-kolla05:46
*** tonanhngo has quit IRC05:48
*** saneax is now known as saneax-_-|AFK05:54
*** iceyao has joined #openstack-kolla05:58
*** iceyao has quit IRC05:59
*** unicell1 has joined #openstack-kolla06:15
*** unicell has quit IRC06:15
*** unicell1 has quit IRC06:22
*** unicell has joined #openstack-kolla06:22
*** tonanhngo has joined #openstack-kolla06:28
*** tonanhngo has quit IRC06:29
*** jbadiapa has joined #openstack-kolla06:30
*** saneax-_-|AFK is now known as saneax06:31
*** jaosorior has joined #openstack-kolla06:31
*** tonanhngo has joined #openstack-kolla07:02
*** david-lyle has quit IRC07:02
*** vhosakot has quit IRC07:03
*** tonanhngo has quit IRC07:03
*** mpansky has joined #openstack-kolla07:04
*** caowei has quit IRC07:08
*** caowei has joined #openstack-kolla07:08
*** sayantan_ has quit IRC07:08
*** Serlex has joined #openstack-kolla07:11
*** unicell has quit IRC07:12
*** unicell has joined #openstack-kolla07:12
*** iceyao has joined #openstack-kolla07:19
*** iceyao has quit IRC07:20
openstackgerritJuan Antonio Osorio Robles proposed openstack/kolla master: RHEL: Add mod_ssl for services running over httpd  https://review.openstack.org/44691107:26
*** manheim has joined #openstack-kolla07:31
*** manheim has quit IRC07:34
*** skramaja_ has joined #openstack-kolla07:34
*** manheim has joined #openstack-kolla07:35
*** skramaja has quit IRC07:37
*** jrobinson has quit IRC07:42
*** tonanhngo has joined #openstack-kolla07:44
*** tonanhngo has quit IRC07:46
*** jmccarthy has joined #openstack-kolla07:47
*** matrohon has joined #openstack-kolla07:51
*** cu5 has quit IRC07:53
openstackgerritMerged openstack/kolla-ansible master: Register Cinder v3 endpoint in keystone  https://review.openstack.org/44688907:53
*** cu5 has joined #openstack-kolla07:57
*** skramaja_ is now known as skramaja08:00
*** shardy has joined #openstack-kolla08:07
*** blallau has joined #openstack-kolla08:08
*** matrohon has quit IRC08:09
*** pcaruana has joined #openstack-kolla08:11
openstackgerritJuan Antonio Osorio Robles proposed openstack/kolla master: RHEL: Add mod_ssl for services running over httpd  https://review.openstack.org/44691108:17
*** blallau has quit IRC08:23
*** shardy has quit IRC08:24
openstackgerritBertrand Lallau proposed openstack/kolla-ansible master: Fix error during Grafana container restart  https://review.openstack.org/44719108:29
openstackgerritDou Rui Yuan proposed openstack/kolla-ansible master: Doc expressions update for Deployment Philosophy and Production architecture  https://review.openstack.org/44688208:29
-openstackstatus- NOTICE: Wiki is broken with database problems, we are working to resolve it08:31
*** ChanServ changes topic to "Wiki is broken with database problems, we are working to resolve it"08:31
*** matrohon has joined #openstack-kolla08:32
*** zhurong has joined #openstack-kolla08:34
*** matrohon has quit IRC08:36
*** ChanServ changes topic to "support: ask.openstack.org | New to Kolla: docs @ http://docs.openstack.org/developer/kolla/ | | Kolla IRC meetngs on Wednesdays @ 16:00 UTC - see agenda @ https://goo.gl/OXB0DL - IRC channel is *LOGGED* @ http://goo.gl/3mzZ7b"08:40
-openstackstatus- NOTICE: Wiki problems have been fixed, it's up and running08:40
*** mgoddard has joined #openstack-kolla08:44
jaosoriormandre: finally figured the issue out with the services trying listen on port 443 and failing.08:46
jaosoriormandre: mod_ssl installs a file with defaults to enable the SSL engine, and it has a Listen directive that attempts to listen there. So, I'm removing that directive, since anyway we don't want to add that unless we explicitly specify such a thing in the configurations.08:47
openstackgerritDennis Mueller proposed openstack/kolla-ansible master: Added neutron metering-agent support  https://review.openstack.org/44585808:49
*** egonzalez has joined #openstack-kolla08:54
openstackgerritJuan Antonio Osorio Robles proposed openstack/kolla master: RHEL: Add mod_ssl for services running over httpd  https://review.openstack.org/44691109:02
*** openstackgerrit has quit IRC09:03
*** athomas has joined #openstack-kolla09:06
*** blallau has joined #openstack-kolla09:15
*** matrohon has joined #openstack-kolla09:29
*** duonghq has quit IRC09:52
*** tovin07_ has quit IRC09:58
*** Serlex has quit IRC09:59
*** Margin has quit IRC10:04
*** mpansky has left #openstack-kolla10:04
*** openstackgerrit has joined #openstack-kolla10:05
openstackgerritJuan Antonio Osorio Robles proposed openstack/kolla master: RHEL: Add mod_ssl for services running over httpd  https://review.openstack.org/44691110:05
*** zhurong has quit IRC10:05
*** cuongnv has quit IRC10:10
*** jaosorior is now known as jaosorior_brb10:14
Daviey_please can i get another +2 on a simple change, https://review.openstack.org/#/c/444771/10:17
openstackgerritBertrand Lallau proposed openstack/kolla-ansible master: Remove ml2_conf.ini from neutron-metadata-agent  https://review.openstack.org/44799210:19
*** satyar has quit IRC10:24
*** zhurong has joined #openstack-kolla10:24
*** manheim has quit IRC10:40
*** shardy has joined #openstack-kolla10:45
*** porzech has quit IRC10:55
*** qiliang28 has joined #openstack-kolla10:58
hrwhi10:58
openstackgerritBertrand Lallau proposed openstack/kolla-ansible master: Remove ml2_conf.ini from some Neutron agents  https://review.openstack.org/44799210:59
*** Serlex has joined #openstack-kolla10:59
*** qiliang27 has quit IRC10:59
*** qiliang28 is now known as qiliang2710:59
*** matrohon has quit IRC11:05
*** iceyao has joined #openstack-kolla11:06
*** eaguilar has joined #openstack-kolla11:06
*** bjolo has joined #openstack-kolla11:07
bjolohowdy peeps11:09
egonzalezsup bjolo11:10
*** iceyao has quit IRC11:10
bjologood11:11
bjolotrying to find my way back to IRC11:11
bjolobeen so busy on actual work, no time for getting distracted on IRC :)11:11
bjolohow are you?\11:11
hrwfine11:12
hrwsomeone knows irc nickname of Paweł Krupa?11:13
egonzalezhrw, no idea11:14
hrwalways worth asking ;D11:14
*** eaguilar_ has joined #openstack-kolla11:23
*** eaguilar has quit IRC11:24
*** manheim has joined #openstack-kolla11:27
*** jmccarthy1 has joined #openstack-kolla11:28
*** jmccarthy has quit IRC11:28
*** rwellum has quit IRC11:29
*** rwallner has joined #openstack-kolla11:31
openstackgerritEduardo Gonzalez proposed openstack/kolla master: Make keystone_user module fully shade  https://review.openstack.org/44801411:31
openstackgerritEduardo Gonzalez proposed openstack/kolla master: Make keystone_user module fully shade  https://review.openstack.org/44801411:34
*** athomas has quit IRC11:44
*** rmart04 has joined #openstack-kolla11:45
*** eaguilar_ has quit IRC11:47
*** eaguilar has joined #openstack-kolla11:48
*** caowei has quit IRC11:53
*** rwellum has joined #openstack-kolla12:02
*** iceyao has joined #openstack-kolla12:06
openstackgerritDou Rui Yuan proposed openstack/kolla-ansible master: Doc expressions update for Deployment Philosophy and Production architecture  https://review.openstack.org/44688212:09
*** eaguilar has quit IRC12:10
openstackgerritZeyu Zhu proposed openstack/kolla-ansible master: Fix the typo  https://review.openstack.org/44803512:12
*** haplo37 has quit IRC12:15
openstackgerritBertrand Lallau proposed openstack/kolla-ansible master: Remove ml2_conf.ini from some Neutron agents  https://review.openstack.org/44799212:17
*** tonanhngo has joined #openstack-kolla12:18
*** srwilkers has joined #openstack-kolla12:18
*** tonanhngo has quit IRC12:19
*** rhallisey has joined #openstack-kolla12:19
*** goldyfruit has joined #openstack-kolla12:19
*** goldyfruit has quit IRC12:20
*** goldyfruit has joined #openstack-kolla12:21
openstackgerritEmilien Macchi proposed openstack/kolla master: Switch trunk/cbs/buildlogs to use https  https://review.openstack.org/44804312:21
*** iceyao has quit IRC12:21
*** mgoddard has quit IRC12:22
*** iceyao has joined #openstack-kolla12:22
*** haplo37 has joined #openstack-kolla12:24
*** iceyao has quit IRC12:25
*** goldyfruit has quit IRC12:25
*** zhubingbing_ has quit IRC12:27
*** iceyao has joined #openstack-kolla12:27
openstackgerritDou Rui Yuan proposed openstack/kolla-ansible master: Doc syntax error and expressions updates  https://review.openstack.org/44688212:28
pbourke_egonzalez: can you have a look at https://review.openstack.org/#/c/435024/12:36
pbourke_egonzalez: and also its dependant patch12:36
*** dave-mccowan has joined #openstack-kolla12:37
mnaserdang12:37
mnasergate still broken?12:37
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible master: Fix wrong api interface reference in haproxy.cfg for nova serial  https://review.openstack.org/44806712:37
*** jaosorior_brb is now known as jaosorior12:38
srwilkersmorning12:39
*** zhurong has quit IRC12:39
*** 21WAAVN4H has joined #openstack-kolla12:39
*** matrohon has joined #openstack-kolla12:42
jaosoriormnaser: are those the failures in centos and ubuntu source?12:44
mnaserjaosorior https://bugs.launchpad.net/kolla/+bug/1674483 yeah12:44
openstackLaunchpad bug 1674483 in kolla "Bifrost failing because of missing SUDO_USER" [Undecided,New]12:44
mnaseri have a few $job things i have to take care of so if someone wants to look into it.. :x12:44
jaosorioroh right, I was wondering what that was12:44
jaosoriorthanks for the info12:45
hrwhm. hope it is that why bifrost-base fail12:45
*** dave-mccowan has quit IRC12:49
*** rhallisey has quit IRC12:49
*** tonanhngo has joined #openstack-kolla12:49
*** rhallisey has joined #openstack-kolla12:51
*** tonanhngo has quit IRC12:51
*** Margin has joined #openstack-kolla12:53
*** leseb has quit IRC12:53
*** kencjohnston_ has quit IRC12:56
*** shardy has quit IRC12:57
*** dave-mccowan has joined #openstack-kolla12:57
*** Margin has quit IRC12:59
*** Margin has joined #openstack-kolla13:00
*** m1dev has joined #openstack-kolla13:01
Marginhello, I opened a bug https://bugs.launchpad.net/kolla/+bug/167467213:02
openstackLaunchpad bug 1674672 in kolla "ironic-conductor doesn't work " [Undecided,New]13:02
*** shardy has joined #openstack-kolla13:03
thomas_oneillMargin: is there anything in your /var/lib/docker/volumes/kolla_logs/_data/ironic* about this?13:05
*** iceyao has quit IRC13:05
Marginno ironic-conductor.log file , only ironic-api.log  ironic-inspector.log13:07
*** tonanhngo has joined #openstack-kolla13:07
thomas_oneillMargin: what happens when you manually stop and start the container13:07
*** kbaegis has joined #openstack-kolla13:08
*** tonanhngo has quit IRC13:08
*** iceyao has joined #openstack-kolla13:11
*** 21WAAVN4H has quit IRC13:15
*** goldyfruit has joined #openstack-kolla13:16
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP Adding virtualbmc to the gate  https://review.openstack.org/44155013:16
Marginno clue after stop and start the container.13:16
*** rhallisey has quit IRC13:20
*** rhallisey has joined #openstack-kolla13:21
thomas_oneillMargin: it doesn't give you any error? the container is still in a restarting state?13:26
thomas_oneillMargin: check systemctl status docker13:26
*** crushil has joined #openstack-kolla13:27
*** jtriley has joined #openstack-kolla13:29
*** mgoddard has joined #openstack-kolla13:31
*** bjolo has quit IRC13:31
*** leseb has joined #openstack-kolla13:32
*** bjolo has joined #openstack-kolla13:34
Marginno clue13:37
*** tonanhngo has joined #openstack-kolla13:39
Marginsudo  " no tty present and no askpass program specified"   is clue ?13:39
*** eaguilar has joined #openstack-kolla13:40
*** adrian_otto has joined #openstack-kolla13:40
*** tonanhngo has quit IRC13:41
*** rwsu has quit IRC13:45
*** cuongnv has joined #openstack-kolla13:48
*** eaguilar has quit IRC13:49
*** deadnull has joined #openstack-kolla13:50
*** manheim has quit IRC13:50
*** manheim has joined #openstack-kolla13:50
*** tovin07_ has joined #openstack-kolla13:55
*** tovin07_ has left #openstack-kolla13:57
*** adrian_otto has quit IRC13:58
*** tovin07_ has joined #openstack-kolla13:58
*** scsnow_ has joined #openstack-kolla14:00
*** tonanhngo has joined #openstack-kolla14:01
scsnow_hi, I'm following kolla-kubernetes quickstart guide. During rabbitmq bootstraping I'm faced with: Failed to pull image "kolla/centos-binary-rabbitmq:auto": Tag auto not found in repository docker.io/kolla/centos-binary-rabbitmq14:02
scsnow_The same for mariadb bootstraping14:02
*** tonanhngo has quit IRC14:02
scsnow_Obviously tag auto doesn't exist, but I didn't find a way to configure it14:03
sbezverkscsnow_: you need to provide some variables, one of them is image tag14:06
*** jtriley_ has joined #openstack-kolla14:07
sbezverkthe easiest approach would be generate cloud.yaml by using build_example_yaml.py14:07
*** rwellum_ has joined #openstack-kolla14:08
sbezverkscsnow_: and then customize it to your needs14:08
*** jtriley has quit IRC14:10
scsnow_sbezverk, where should I put generated cloud.yaml?14:10
*** satyar has joined #openstack-kolla14:12
*** lamt has joined #openstack-kolla14:27
*** bjolo has quit IRC14:31
*** tonanhngo has joined #openstack-kolla14:33
*** tonanhngo has quit IRC14:33
*** fguillot has joined #openstack-kolla14:34
*** kbaegis has quit IRC14:39
*** kbaegis has joined #openstack-kolla14:42
*** shashank_t_ has joined #openstack-kolla14:47
*** jtriley_ has quit IRC14:49
*** david-lyle has joined #openstack-kolla14:49
*** lamt has quit IRC14:52
*** jtriley has joined #openstack-kolla14:52
*** lamt has joined #openstack-kolla14:54
sbezverkscsnow_: location is not important, as long as you provide path in --values14:56
sbezverkto it.. example I use helm install blah blah --values examples/cloud.yaml14:57
*** blallau has quit IRC15:00
*** dave-mccowan has quit IRC15:02
*** blallau has joined #openstack-kolla15:03
*** Margin has quit IRC15:04
*** lamt has quit IRC15:08
*** david-lyle has quit IRC15:09
*** kbaegis has quit IRC15:09
openstackgerritMerged openstack/kolla-ansible master: Update Fluentd config to log in Kibana dashboard  https://review.openstack.org/44634215:13
*** david-lyle has joined #openstack-kolla15:15
*** manheim has quit IRC15:18
inc0good morning15:18
*** manheim has joined #openstack-kolla15:19
srwilkersmorning15:19
openstackgerritMerged openstack/kolla-ansible master: Fix logrotate issue on Octavia and Watcher service  https://review.openstack.org/44606115:19
openstackgerritOpenStack Proposal Bot proposed openstack/kolla-ansible master: Updated from global requirements  https://review.openstack.org/44677215:21
*** dave-mccowan has joined #openstack-kolla15:23
*** shardy is now known as coding_shardy15:23
*** tonanhngo has joined #openstack-kolla15:23
*** manheim has quit IRC15:23
*** tonanhngo has quit IRC15:24
*** sayantani01 has joined #openstack-kolla15:27
*** sayantani01 has quit IRC15:29
*** matrohon has quit IRC15:29
*** sayantani01 has joined #openstack-kolla15:29
sbezverkinc0: srwilkers: good morning15:30
sbezverksrwilkers: has portdirect talked to you about tomorrow? possibility to have a chat about our session?15:31
*** jrist has quit IRC15:33
*** hieulq_ has joined #openstack-kolla15:33
*** hrw has quit IRC15:35
*** hrw has joined #openstack-kolla15:36
*** adrian_otto has joined #openstack-kolla15:38
*** jrist has joined #openstack-kolla15:40
inc0sbezverk: question15:42
inc0https://github.com/openstack/kolla-kubernetes/blob/296fd38507cb0c5a487383db5bf330214de977c8/tools/setup_gate_ceph.sh#L140 what are configmaps -haproxy?15:42
inc0where do they come from?15:43
*** zhubingbing has joined #openstack-kolla15:44
inc0ahh...I see it now15:44
inc0it's in helm15:44
*** cuongnv has quit IRC15:49
*** pbandark has joined #openstack-kolla15:49
jmccarthy1Hmm I notice services seem to have this pattern with transport_url like transport_url = rabbit://{% for host in groups['rabbitmq'] %}... which puts a list of related hosts into services config - why is this ? Or for example why not use haproxy ? i.e. point to VIP and let it deal with the rest ?15:50
*** blallau has quit IRC15:52
*** adrian_otto has quit IRC15:52
*** tonanhngo has joined #openstack-kolla15:53
*** pbandark has quit IRC15:53
*** jmccarthy1 has quit IRC15:54
*** david-lyle has quit IRC15:54
*** jmccarthy has joined #openstack-kolla15:54
*** foliouno has joined #openstack-kolla15:57
sbezverkinc0: we have configmap kind of hardcoded15:57
sbezverkI mean only for haproxy15:57
sbezverkif you check services/common you will find a template we use for haproxy config map15:58
sbezverkinc0: I do not think we moved them to helm, at least I do not rememeber ;)15:58
inc0so I don't need to create this configmap from /etc/kolla right?15:59
sbezverkinc0: unfortunately we do need them, but they will not come from kolla15:59
inc0sbezverk: some of them are there https://github.com/openstack/kolla-kubernetes/blob/8652282f7b32e59965863e07cad7aad511a568f3/helm/test/devenv/templates/nova-api-haproxy.yaml16:00
inc0how do I generate them?16:00
sbezverkinc0: please ignore those, they were generated manually just for dev env16:00
*** david-lyle has joined #openstack-kolla16:01
sbezverkinc0: they are all hardcoded with values which is completely useless for general deployment16:01
*** adrian_otto has joined #openstack-kolla16:01
*** magicboiz has joined #openstack-kolla16:01
inc0sure...so how do I generate real ones?16:01
sbezverkinc0: you need to run kollakube tools16:02
inc0so here's thing...I want to get rid of kollakube16:02
inc0alltogether16:02
inc0if we have python generated configs in same place as ansible generated configs, it's worse than I thought16:02
inc0:(16:02
sbezverkfor haproxy kollakube checks etc/kolla-kubernetes/service_resource.yaml16:02
sbezverkwhere haproxy config maps are defined16:03
spsuryainc0: morning16:03
inc0good morning spsurya16:03
sbezverkand refer to a template16:03
*** lamt has joined #openstack-kolla16:03
inc0ok, so I have slightly different idea16:03
sbezverkinc0: yeah we have mixture now16:03
inc0I need to warn you, it does involve ansible (but not kolla-ansible)16:03
sbezverkinc0: have you sold it to kfox1111 ;)?16:04
spsuryainc0:  it would be great if you review this   again https://review.openstack.org/#/c/425446/16:04
inc0please, let's put religion aside16:05
inc0we need to move forward without this burden16:05
*** lamt has quit IRC16:05
inc0if only we agree on having ansible in kolla-k8s, at least for time being16:05
inc0I can give us separatied configs today16:05
sbezverkinc0: sure, but in this case when some folks will start saying that we are moving away from helm native way, you will need to take heat for it ;)16:06
inc0single good way to create all the configs with no kolla-ansible involved16:06
*** lamt has joined #openstack-kolla16:06
inc0well, fuck that16:06
inc0pardon my language16:06
sbezverkinc0: how do you connect helm/all_values.yaml with ansible generating configmaps?16:07
inc0negactive16:07
inc0negative16:07
inc0I'll just copy ansible dir from kolla-ansible and thin it out to only have configmap16:07
*** rmart04 has quit IRC16:08
sbezverkinc0: ok but helm/all_values.yaml has values which will be needed for configmap generation16:08
inc0then we can add all the k8s specific configs like this -haproxy crap16:08
inc0ok...well...they're not used now16:08
inc0but anyway, can do it16:09
inc0it's all yaml and ansible understands yaml16:09
sbezverkinc0: currently we have two stages16:09
inc0kfox1111: around?16:09
sbezverkwe create configmaps with some unknown values and second stage gets populated congmaps with "running" info by using init containers16:10
Daviey_please can i get another +2 on a simple change, https://review.openstack.org/#/c/444771/16:10
inc0right because we don't have host specific stuff16:10
sbezverkinc0: so If I undersatnd you right, you plan to replace only the first stage16:10
inc0sbezverk: for start at least16:11
sbezverkinc0: ok16:11
inc0but first I need kfox1111 to tell me that he won't go on religious war with me if I do it in ansible that will be optional16:11
Daviey_inc0: Regarding ansible, i'm assuming you are talking about laying down /etc/kolla on the remote hosts?  Surely the cleaner fix is to move this to a docker volume... then ansible isn't needed?16:12
sbezverkinc0: :) yeah curious to see this debate16:12
Daviey_(and would be cleaner for kolla-ansible aswell)16:12
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla-kubernetes master: WIP: Playbook to add configmaps and secrets  https://review.openstack.org/44773316:12
inc0Daviey_: issue we're trying to deal with is how to generate config files for kolla-k8s16:14
*** gfhellma has joined #openstack-kolla16:14
inc0currently it's done with abominable hack16:14
Daviey_inc0: right... but if we move it to a docker volume.. then it becomes cleaner for kolla-ansible and works for kolla-k8s aswell?16:14
inc0so I really don't want to drive our arch on premise that "some people are alergic to ansible"16:14
openstackgerritMerged openstack/kolla-ansible master: Remove heka_socket vol and unwedge some containers  https://review.openstack.org/44477116:15
inc0Daviey_: well, yes and no, they still need to be created16:15
Daviey_(for reference, i've been pondering a kolla-nomad project.. but the config problem is what has been bugging me)16:15
inc0and docker volume is just a directory by default16:15
inc0all you need to do Daviey_ is to mount them to container somehow16:15
Daviey_inc0: Right... but as a volume we have less host dependency16:16
inc0it can be bindmount, it can be configmap it can be docker volume16:16
Daviey_Both k8 and Nomad support a notion of artifacts... So all the controller server needs to run is a http server where they can be retrieved16:16
Daviey_The artifacts is a tarball of /etc/kolla16:16
Daviey_shove that straight into a docker volume, and make it available to multiple containers16:16
inc0Daviey_: that'd work for nomad, sure16:17
Daviey_inc0: k8's can have artifacts aswell16:17
openstackgerritOpenStack Proposal Bot proposed openstack/kolla-ansible master: Updated from global requirements  https://review.openstack.org/44677216:17
openstackgerritMerged openstack/kolla-ansible master: Fix wrong api interface reference in haproxy.cfg for nova serial  https://review.openstack.org/44806716:17
inc0my point is16:17
inc0you still need to create file itself:)16:17
inc0how you mount it to container - that depends on orch engine16:17
inc0with k8s it's configmap, with nomad it can be served over http16:18
Daviey_inc0: At the moment, ansible is creating the initial rendering of config right.. for everything?  Is ansible the right tool for this?16:19
inc0know any better?16:19
Daviey_I'm not suggesting we re-invent the world... but as a simple rendering problem.. maybe a kolla standalone tool for rendering templates.. that is then used by kolla-ansible, kolla-k8's and kolla-anythingelse ?16:20
*** hieulq_ has quit IRC16:20
inc0Daviey_: https://review.openstack.org/#/c/447733/216:20
inc0here I use ansible as just a rendering toolk16:21
Daviey_kolla-renderconfig --output configmap ?16:21
inc0much easier than writing my own from scratch16:21
Daviey_fair point16:21
inc0sbezverk: you know what's funny?16:22
zhubingbingaround Bertrand Lallau16:22
inc0today I thought16:22
openstackgerritMerged openstack/kolla-ansible master: Fix Magnum Kolla demo in contrib  https://review.openstack.org/44234716:22
inc0if we'd just use ansible as templating tool instead of helm16:22
inc0we'd be done with kolla-k8s by now16:22
*** tovin07_ has quit IRC16:23
*** kbaegis has joined #openstack-kolla16:23
sbezverkinc0: well we had k8s running with jinja16:23
Daviey_If we used Django as a templating engine.. we'd also might be done.. Doesn't mean it is graceful :)16:23
inc0ansible is more than just jinja16:23
bmacesdake: sorry i missed your ping yesterday evening. whats up?16:24
inc0Daviey_: ansible gives you both pieces we need - templating and orchiestration16:24
openstackgerritMerged openstack/kolla-ansible master: Fix Telegraf startup config files issue  https://review.openstack.org/44738216:24
inc0k8s can just consume resources templated16:24
sbezverkinc0: then community decided to go with helm... I do not think going back or ansible will be taken lightly especially after all pain we went with switching to helm..16:24
inc0but whatever, this ship has sailed16:24
*** Mech422 has joined #openstack-kolla16:24
Daviey_inc0: But does that sound like a clean implementation of k8's?16:24
inc0nah, I'm not suggesting removing helm and going back to ansible16:24
inc0Daviey_: why not?16:24
inc0you'll be closer to native k8s than helm imho16:25
inc0as ansible will just create resource definition for you16:25
Mech422SamYaple: I saw your email about rabbitmq, which brought up something I've been wondering about: if OpenStack survives/deals with the occasional lost message - why bother with rabbitmq at all ?16:26
Daviey_inc0: I eat my words.. it isnt so bad.. https://github.com/mxabierto/k8-ansible16:26
sbezverkinc0: we need rendering engine, there is no way around it and we need packaging.. so far helm seems doing its job..16:26
inc0sbezverk: what do you mean by "packaging"?16:26
inc0Daviey_: it would be better if it would be proper ansible;)16:27
inc0anyway, again, this is academical by now16:27
inc0I just know that when I'll write new code for k8s16:27
inc0I'll use ansible for it16:27
inc0I think ansible would do great as "orchiestrator" and templating engine for k8s16:28
inc0rbergeron: ^16:28
*** jaosorior has quit IRC16:29
Mech422inc0: btw - saw the discussion about adding etcd to provisioning ... we actually just decided to do somethign similiar for our in-house stuff using consul16:29
Daviey_inc0: I massively dislike the fire-and-forget of kolla-ansible and the suck-it-and-see approach to changing stuff.. Scares the crap out of me.  That is why i like Nomad, as it is pretty workfow independent.. but you can easily see the changed resources before running.16:30
inc0Mech422: we had some discussion on PTG to use etcd for configs...is that what you're reffering to?16:30
inc0Daviey_: the way I'd do it is to ask ansible to create bunch of resource files16:31
Mech422inc0: yeah - though we're prolly gonna end up doing more then configs in-house16:31
inc0and feed them to k8s on my won16:31
inc0configs in house are way to go16:31
inc0kolla-ansible can help you but really if you can handle hand-crafting configs16:32
inc0that's how I'd do it16:32
Mech422inc0: big thing that drove the conversation for us was 'e-brake' functionality for our Salt stuff - stopping a upgrade and doing rollbacks if a canary deploy fails16:32
inc0that's good feature to have16:33
Daviey_Mech422: But you trust the rollbacks?16:33
Mech422Daviey_: umm - above my paygrade ? :-P16:34
Daviey_Mech422: I mean, with k8.. you can bring up a new container... show you are happy.. then divert traffic to it.  The rollback is to divert traffic back to the old containers16:34
Mech422Daviey_: we're charging each product with supplying rollback formulae as part of the production turnover....16:34
Daviey_CloudFormation has bitten me so many times, with failed rollbacks and you are left in a state that the tool can't fix16:35
Mech422inc0: anyway - sounds like it would be a good feature to have... lots of stuff seems to want/need it..16:35
Mech422inc0: having a 'standard' one seems like a good thing16:35
Daviey_Terraform, will just go "oh crap" i broke.. but leave the state defintion of what the current world is.. and allow you to fix it and keep reapplying your config to "roll-forward".. i prefer this model myself16:36
inc0Mech422: it's ansible, ansible offers lots of these stuff16:36
Daviey_ansible has "check mode"... to see what it plans to change, but when i last tried it with kolla-ansible it didn't work16:37
Mech422Daviey_: yeah - we looked at Terraform for some stuff - but the 'state dir' seemed fragile, and we were pretty sure SysOps would end up making changes that broke state16:37
Mech422Daviey_: "oh,I'll just shut this machine down since its idle" sort of stuff16:37
Daviey_Mech422: I'm storing the state file in s3, with auto-versioning on s3-side.  Hasn't been a problem yet.16:37
sbezverkgents I am debugging baremetal issue I see at the gate and I need to collect baremetal vm logs, (it is regular virsh defined vm) by some reason I cannot get bios related messages and messages which are generated even before os gets loaded, any one has experience in this?16:38
Daviey_sbezverk: Ask #openstack-infra, they might be willing to pull it for you16:38
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla-kubernetes master: WIP: Playbook to add configmaps and secrets  https://review.openstack.org/44773316:38
inc0sbezverk: ^16:39
sbezverkDaviey_: it is not that VM they give us for running a gate job16:39
Mech422Daviey_: thats good to know.. we were worried about state, and we also liked the fact that consul/etcd let us 'share' the knowledge easily - lots of stuff can talk to them, instead of it being 'locked up' in salt pillars16:40
sbezverkDaviey_: it is VM I defined and use for simulating baremetal provisioning16:40
sbezverkinc0: how this approach would be extensible? example I have a couple of services which are not yet merged like nova placement? would it require adding new service to the list to get configmap generated?16:42
Daviey_sbezverk: but if you can't see something.. and having out of band ssh to the box allows you to see it.. they might pull it out for you16:43
Daviey_sbezverk: Are you dumping the qemu console out to txt?  Or just the tty console (after the kernel has started)?16:44
sbezverkDaviey_: I mapped console to file16:45
Daviey_Mech422: ah yes, i've not used the consul support for TF state.. it made me a bit nervous aswell.  S3 being flat static versioned file made me feel safer.16:45
Daviey_sbezverk: right, but which console?  tty/kernel or the 'server' console (which would include the bios output)16:46
Mech422Daviey_: oh - we're not (planning on?) using Terraform atm.16:46
Mech422Daviey_: first we're just going with a reliable storage setup for 'state-like' stuff16:47
Mech422Daviey_: we may revisit TF later though16:47
Daviey_Mech422: so what are you using now?16:47
rbergeroninc0: aye -- i assume you have seen ansible-container -- https://github.com/ansible/ansible-container (may not be exactly what you're looking for) -- but there are kubernetes modules already, and someone was also working on (though i haven't yet seen) a helm module for ansible16:47
rbergeroninc0: https://github.com/kubernetes-incubator/kargo/issues/661#issuecomment-26547406316:48
rbergeroninc0: just depends on how you want to abstract the abstractions and orchestrations of orchestrators :)16:48
*** Daviey_ is now known as Daviey16:49
Mech422Daviey_: for most stuff - just plain Salt16:49
*** bmace has quit IRC16:49
DavieyMech422: ah, cool16:49
*** bmace has joined #openstack-kolla16:49
sbezverkDaviey_: in my local lab I connect with VNC to that VM and see everything even bios related messages. at the gate I do not have luxury of VNC but I need to see bios related messages as I suspect the issue is before OS gets loaded16:49
Daviey-> #o'-infra16:50
rbergeroninc0: also, iirc flaper87 may also be looking at such things16:50
*** crushil has quit IRC16:51
*** satyar has quit IRC16:51
* flaper87 looks at many things (including the backlog of this conversation)16:51
*** jaosorior has joined #openstack-kolla16:55
*** coding_shardy is now known as shardy16:57
*** kbaegis has quit IRC17:00
sdakehey peeps17:01
sdakeinc0 we made a decision as a community to dump ansible workflows in kolla-kubernetes17:02
sdakeinc0 that was mostly driven by you :)17:03
*** egonzalez has quit IRC17:04
*** cu5 has quit IRC17:05
*** yee379 has quit IRC17:06
*** yee379 has joined #openstack-kolla17:06
*** cu5 has joined #openstack-kolla17:06
*** kbaegis has joined #openstack-kolla17:07
*** vhosakot has joined #openstack-kolla17:09
*** srwilkers has quit IRC17:10
*** unicell has quit IRC17:10
*** skramaja has quit IRC17:12
sdakebmace i wanted to know if you could review https://review.openstack.org/#/c/447356/17:15
sdakeinc0 helm is a package manager - that is how it is being  treated for the most part - it may grow workflow features in the future17:16
sdakeinc0 first we want to get to helm native packaging of all the stuff - and then sort out the workflow natively17:16
sdakealot of the workflow is driven by how helm packages stuff up17:17
bmacesdake: sure, will look at it in a few17:17
kfox1111morning.17:17
sdakethat packaged up bits we are missing is "config maps"17:17
sdakesup kfox111117:17
sdakeinc0 creating configmaps != workflow17:17
sdakebmace thanks - hopefully plan to spin that document today with eveyrone's comments fixed17:18
*** zhubingbing has quit IRC17:22
*** srwilkers has joined #openstack-kolla17:22
sdakesrwilkers o/17:22
sbezverkkfox1111: good morning17:25
sbezverksdake: good morning17:25
*** pbourke_ is now known as pbourke17:25
openstackgerritOpenStack Proposal Bot proposed openstack/kolla-ansible master: Updated from global requirements  https://review.openstack.org/44677217:26
sdakesup sbezverk17:26
sdakeI got a late start today ;(17:26
sdakeall that travel caught up with me I htink :)17:26
sbezverkkfox1111: gate job driving me mad :( the same damn code work 100% in my test bed, but still failing at the gate.17:26
sbezverkkfox1111: vm does not call back to ironic-api17:27
*** kbaegis has quit IRC17:27
kfox1111sbezverk: hmm...17:27
kfox1111iptables?17:27
*** kbaegis has joined #openstack-kolla17:27
kfox1111try turning it off.17:27
sdakei think flushing the rules first makes more sens ethen turning it off - I suspect but odn't knwo that k8s and openstack use iptables in some way17:28
kfox1111sdake: yeah. thanks. that s what I meant.17:28
kfox1111flush it at the very begining of the gate job, then see if it works better.17:28
sdakesbezverk iptables -f I think does that17:28
kfox1111I know they have some very custom iptables rules on the vms.17:29
srwilkershey sdake o/17:29
*** lamt has quit IRC17:29
sdakesbezverk although best to reference stackoverflow17:30
sdakeeverytime i want to flush rules, i have to look it up:)17:30
*** zhubingbing has joined #openstack-kolla17:31
sbezverkkfox1111: I finally get config to log all VM messages even when it is in bios17:32
kfox1111sbezverk: awesome. :)17:32
kfox1111what did you do?17:32
sbezverkhopefully that could help to identtify at what point it get stuck17:32
sbezverkkfox1111: there is an option in virsh17:32
kfox1111ah. cool.17:32
sbezverkkfox1111: <bios useserial='yes'/>17:33
*** tonanhngo has quit IRC17:33
kfox1111nice.17:33
sbezverkkfox1111: if not use then only OS messages are logged and since I was not reaching OS stage, nothing got logged17:33
kfox1111yeah. that makes sense.17:34
*** shashank_t_ has quit IRC17:35
inc0sdake: I'm only talking of configmaps tnow17:36
*** rwsu has joined #openstack-kolla17:36
inc0now17:36
*** david-lyle_ has joined #openstack-kolla17:37
inc0kfox1111: sbezverk where are files that genconfig created in gates?17:37
inc0files normally would land in /etc/kolla17:37
kfox1111 /etc/kolla17:37
sdakecopying over the genconfig code from kolla-ansible and landing just those bits would make the ejection of kolla-ansible as a dep go faster17:37
sdakeas in immediate17:37
inc0yup17:38
inc0that's what I wanted to doscuss17:38
sdakegenpwd needs to be copied17:38
sdakeas well17:38
*** david-lyle has quit IRC17:38
sdakei was planning to do that this week when i got to it :)17:38
inc0I can make this patchset probably...idk today? without gates17:38
sdake(genpwd copying)17:38
kfox1111hmm... so, fork genconfig and strip it?17:38
sdakekfox1111 that is the proposal on the table :)17:38
inc0kfox1111: exactly17:38
kfox1111interesting.17:38
inc0add all the missing templates17:38
kfox1111that should be very quick to do, and greatly simplify things...17:39
*** unicell has joined #openstack-kolla17:39
inc0combine with with logic to create k8s resources instead of only configs17:39
sdakeits a winner17:39
inc0kk I'm on it17:39
sdakewe still need external config maps after tha tstep :)17:39
kfox1111+117:39
*** eaguilar has joined #openstack-kolla17:39
inc0well it won't on it's own solve per-node overrides17:39
sdakeejecting kolla-ansible as a dep woudl greatly simplify kolla-kubernetes dev17:39
sdakeya per node overrides can be done with helm configmaps I am pretty sure17:40
inc0we need to bake this into helm configmap mounting logic17:40
inc0so we can *create* per node configmap with ansible17:40
sdakejust add multiple config maps17:40
inc0to some extend17:40
*** EmilienM has joined #openstack-kolla17:40
sdakeand run nova config1 config2 etc17:40
inc0as in, if we provide all the interface/ips and stuff17:40
EmilienMsdake: the -1 sounds unfair on https://review.openstack.org/#/c/448043 tbh17:41
EmilienMinc0: ^17:41
sdakeEmilienM i didn't -1 it :)17:41
sdakeEmilienM moment17:41
* EmilienM trying to be helpful17:41
inc0EmilienM: gates are red...17:41
EmilienMsdake: I know but caoyuan did, not sure if this person is here17:41
EmilienMinc0: yeah no worries17:41
sdakeEmilienM lemme recheck it17:42
EmilienMI just found useful to show this kind of useless -1 review17:42
inc0and that was his comment, so comment was...redundant;)17:42
inc0yeah, I'll add this to review17:42
sdakeEmilienM even though our gates are nonvoting we do want them to be green :)17:42
inc0voting gates are red too17:42
inc0centos source namely17:42
sdakeinc0 the gates are jammed up atm I believe17:42
sdakeinc0 the ironic bug17:43
kfox1111inc0: your very concerned with per node config. but k8s is all about not having to do that.17:43
sdakewith inspector17:43
sdakewrather with biforst17:43
sdaketoo many components :)17:43
inc0kfox1111: well...neutron still needs it's interfaces17:43
inc0and provider_network still needs it's vlan tags17:43
kfox1111inc0: yes, but you very very veary rarely have a situation where you have every machine needs a different interface.17:43
inc0not to mention all sorts of host aggregatey stuff17:43
kfox1111usualy its a few dozen hardware types. or less.17:44
sdakeEmilienM change looks good to me17:44
sdakeEmilienM i think our gates are temporarily broken17:44
inc0kfox1111: yeah, usually it's "just" 2 or 3;)17:44
kfox1111so, a k8s host aggregate like thing, not a per host thing.17:44
EmilienMyep, I figured, no worries17:44
sdakeEmilienM lemme dig into the gate and see what is busted17:44
kfox1111so, deploy multiple instances of nova-openvswitch-agent with node selectors and a configmap.17:44
sdakeEmilienM (if its still the bifrost thing)17:44
inc0kfox1111: so create configmap1, deploy on aggregate 1, configmap2 deploy on aggregate 217:45
kfox1111inc0: yeah.17:45
inc0let me put it this way, config overrides per agregate wfm17:45
inc0then you have configmap-aggregate1 and helm/k8s will mount it17:45
kfox1111right.17:46
*** Serlex has quit IRC17:46
inc0that's totally fine17:46
inc0then in ansible you can create configmaps per aggregate17:46
kfox1111right.17:46
inc0for example -e aggregate=aggregate117:46
inc0cool, totally doable then17:46
kfox1111though I don't think we want to spend a lot of time making that work very smoothly before doing it in helm.17:47
sdakeEmilienM enjoy :)17:47
inc0kfox1111: thing is, I think we will already have this mechanism17:47
kfox1111helm install neutron-openvswitch-configmap --set interface=eth0,name=aggregate1 or something like that.17:47
inc0with our current kolla-ansible arch;)17:47
kfox1111ah. ok.17:47
inc0maybe with one more line per task17:47
sdakei guess that is a bit of scope creep for 1.0.017:48
inc0then you can create variable "kubernetes_aggregates: ag1, ag2"17:48
*** iceyao has quit IRC17:48
inc0sdake: it's few hrs of work17:48
inc0and we get whole override mechanism17:48
sdakeyou indicated you wouldn't get married to the code ;)17:48
inc0its *much* easier than writing config mechanism to helm17:48
sdakewhen we go to helm i don't think it will be a few hours of work17:48
inc0my point exactly17:49
*** tonanhngo has joined #openstack-kolla17:49
inc0helm might be bad tool for this task17:49
sdakedisagree ;)17:49
sdakealthough ejecting kolla-ansible is good objective17:49
inc0anyway17:49
inc0if we have fully feature complete config generation in kolla-k8s17:49
inc0if someone wants to go lenghts to rewrite it to helm17:49
inc0fine by me17:49
inc0but I'd change 1.0 scope to remove requirement of this being in helm17:50
sdakecool - as long as your not married to it like I said :)17:50
inc0I'm always about what makes sense17:50
*** tonanhngo has quit IRC17:50
inc0I have no emotional attachments to any code17:50
*** zhubingbing has quit IRC17:50
sdakelies :)17:50
inc0and I'm slowly startking to challenge helm being good tool for this job17:51
inc0but that's me17:51
inc0I'm entitled to have my own opinions;)17:51
sdakeagreed17:51
kfox1111yup.17:51
kfox1111thats why we did microservices.17:51
kfox1111to allow anyone to do config, however is best for them.17:51
kfox1111if ansible's the right solution for you, great.17:51
inc0beautiful thing about moving this ansible to kolla-k8s17:51
kfox1111if its helm, or chef or shell, great.17:51
inc0is that it's totally optional17:52
inc0all we need is config files in a dir17:52
inc0and we ship one way to generate them17:52
inc0namely ansible17:52
*** satyar has joined #openstack-kolla17:52
inc0this approach is acceptable?17:52
*** crushil has joined #openstack-kolla17:53
inc0that will also make poritng good changes from kolla-ansible to kolla-k8s much easier17:53
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP Adding virtualbmc to the gate  https://review.openstack.org/44155017:54
sbezverkinc0: guys, are you kidding we spent whole cycle helming everything and now you want to change again!?!?!?!17:56
*** sayantani01 has quit IRC17:56
kfox1111I'm ok with it as a transition thing. as of long term, less convinced.17:57
kfox1111config generation is very religious.17:57
*** sayantani01 has joined #openstack-kolla17:57
kfox1111sbezverk: only talking configmaps.17:57
*** jaosorior has quit IRC17:57
bmacesdake: so, is that doc going to get a lot of massage?  it is kind of a mess right now.17:58
sbezverkkfox1111: ok if it is just configmaps, it scared me, so much time invested into helm..17:58
sdakebmace its getting a full spa treatment after you review it :)17:58
*** srwilkers has quit IRC17:58
*** rwsu has quit IRC17:59
bmacesdake: kk, it is just a little chaotic at this point to get a feel for what really needs to be fixed.. depending on what you are going to yank out completely, which i hope is a lot, like all the in-line comments, etc?17:59
inc0kfox1111: yeah...well...again optional;)18:00
inc0anyway. it's quck win18:00
inc0quick win18:00
inc0so let's take it18:00
*** rwsu has joined #openstack-kolla18:00
inc0kfox1111: http://logs.openstack.org/33/447733/3/check/gate-kolla-kubernetes-deploy-centos-binary-2-ceph-multi-nv/7aaa270/console.html#_2017-03-21_17_00_24_616166 you sure config files are in /etc/kolla/?18:00
*** lamt has joined #openstack-kolla18:01
*** scsnow_ has quit IRC18:01
*** mgoddard has quit IRC18:01
kfox1111sbezverk: yeah. not backing out of helm. its really the right solultion I think for the microservices.18:03
inc0yeah, I agree18:03
*** zhubingbing has joined #openstack-kolla18:04
kfox1111inc0: https://github.com/openstack/kolla-kubernetes/blob/master/tests/bin/gate_capture_logs.sh#L10118:04
kfox1111inc0: http://logs.openstack.org/33/447733/3/check/gate-kolla-kubernetes-deploy-centos-binary-2-ceph-multi-nv/7aaa270/logs/kolla/18:05
kfox1111copyied verbatium into the g ate logs, for your inspection. :)18:05
inc0hmm18:05
*** scsnow_ has joined #openstack-kolla18:06
inc0right...18:06
openstackgerritKevin Fox proposed openstack/kolla-kubernetes master: WIP: Fernet Token Support  https://review.openstack.org/44627418:07
inc0kfox1111: do we copy kubectl describe po too?18:09
kfox1111yes18:09
kfox1111http://logs.openstack.org/33/447733/3/check/gate-kolla-kubernetes-deploy-centos-binary-2-ceph-multi-nv/7aaa270/logs/pods/18:09
kfox1111anything with a -containername.txt is logs form the container,18:09
kfox1111and the other is the describe pod.18:09
kfox1111so, like: http://logs.openstack.org/33/447733/3/check/gate-kolla-kubernetes-deploy-centos-binary-2-ceph-multi-nv/7aaa270/logs/pods/kolla-ceph-rbd-351jn-main.txt18:10
kfox1111logs for the container.18:10
kfox1111http://logs.openstack.org/33/447733/3/check/gate-kolla-kubernetes-deploy-centos-binary-2-ceph-multi-nv/7aaa270/logs/pods/kolla-ceph-rbd-351jn.txt describe for the pod18:10
*** klindgren has quit IRC18:10
inc0ahh...kollakube does secrets too?18:10
kfox1111kollakube did it all, pre helm.18:10
kfox1111there is still a few non configmap things in it.18:11
inc0https://review.openstack.org/#/c/447733/3/tools/setup_gate_ceph.sh18:11
inc0here, gates failed because of no secret created18:11
kfox1111which secret?18:12
inc0http://logs.openstack.org/33/447733/3/check/gate-kolla-kubernetes-deploy-ubuntu-binary-2-ceph-nv/1ebe874/logs/pods/kolla-mariadb-init-element-n228k.txt18:12
inc0nvm18:12
inc0I misread it18:12
kfox1111k. :)18:12
inc0riiiiight18:13
inc0I know what happened18:13
inc0--namespace kolla :(18:13
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla-kubernetes master: WIP: Playbook to add configmaps and secrets  https://review.openstack.org/44773318:14
bmacesdake: i don't see anything super obvious.. would love to see a post-spa version of it and then perhaps when it is more readable it will be easier to find issues.18:14
*** scsnow_ has quit IRC18:18
*** shardy has quit IRC18:18
*** manheim has joined #openstack-kolla18:20
*** tonanhngo has joined #openstack-kolla18:21
*** tonanhngo has quit IRC18:21
jascott1kfox1111 ping18:22
kfox1111jascott1: pong18:22
jascott1hey whats up18:22
jascott1got an import values question18:22
kfox1111doing good. you?18:22
kfox1111sure18:22
jascott1great, signing papers today for house im buying!18:23
kfox1111congrats. :)18:23
jascott1thanks!18:23
dimsjascott1 : nice!18:23
jascott1so yeah18:23
jascott1thanks dims18:24
jascott1wrt to import values18:24
jascott1i dont think we talked about overwriting18:24
jascott1if parent value already exists18:24
jascott1we use imported or parent?18:24
kfox1111parent always wins.18:24
kfox1111and users values wins over parent.18:25
jascott1right18:25
*** manheim has quit IRC18:25
jascott1yeah I didnt do that18:26
jascott1but will fix up in jiffy18:26
kfox1111ah. ok. cool.18:26
jascott1idk what i was thinking18:26
kfox1111no wories. :)18:26
jascott1hey did you get your data center real world problems all sorted?18:27
kfox1111limping along stablily now. :)18:28
*** jmccarthy has quit IRC18:28
jascott1nice18:28
kfox1111we're maybe a week or two, or alternately months from getting that ups back.18:28
*** zhubingbing has quit IRC18:28
*** mgoddard has joined #openstack-kolla18:37
*** pcaruana has quit IRC18:39
*** rwsu has quit IRC18:42
*** haplo37 has quit IRC18:44
*** david-lyle_ has quit IRC18:46
*** haplo37 has joined #openstack-kolla18:49
*** eaguilar has quit IRC18:52
*** lamt has quit IRC19:06
*** shashank_t_ has joined #openstack-kolla19:07
*** tonanhngo has joined #openstack-kolla19:09
*** tonanhngo has quit IRC19:10
*** satyar has quit IRC19:10
*** rwsu has joined #openstack-kolla19:12
*** rwsu has quit IRC19:17
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP Adding virtualbmc to the gate  https://review.openstack.org/44155019:25
inc0kfox1111: parent wins over child?19:27
inc0hmm...I'd rather go child winds over parent and don't add defaults unless need to19:28
inc0like, neutron image == 3.0.2 where all other images == 3.0.319:28
inc0but I guess that could be done by redefinition of new variable19:29
*** MasterOfBugs has joined #openstack-kolla19:36
*** kbaegis has quit IRC19:37
*** kbaegis has joined #openstack-kolla19:38
*** kbaegis has quit IRC19:40
*** kbaegis has joined #openstack-kolla19:41
*** kbaegis has quit IRC19:43
*** mgoddard has quit IRC19:44
*** kbaegis has joined #openstack-kolla19:44
*** rhallisey has quit IRC19:45
*** rwallner has quit IRC19:46
*** rwallner has joined #openstack-kolla19:47
*** tonanhngo has joined #openstack-kolla19:47
*** rhallisey has joined #openstack-kolla19:48
*** tonanhngo has quit IRC19:49
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla-kubernetes master: WIP Move config generation to kolla-k8s  https://review.openstack.org/44825819:50
inc0kfox1111: sbezverk ^19:50
inc0just neutron19:50
inc0but check approach19:50
*** MasterOfBugs has quit IRC19:52
*** lamt has joined #openstack-kolla19:53
*** kbaegis has quit IRC19:55
*** kbaegis has joined #openstack-kolla19:57
*** kbaegis has quit IRC20:00
*** dave-mccowan has quit IRC20:02
*** manheim has joined #openstack-kolla20:02
*** matrohon has joined #openstack-kolla20:02
sbezverkkfox1111: could you check this error, while centos is always passing this point, ubuntu always fails http://logs.openstack.org/50/441550/102/experimental/gate-kolla-kubernetes-deploy-ubuntu-source-3-ironic-nv/4bdd216/console.html#_2017-03-21_19_47_33_85445920:03
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla-kubernetes master: WIP: Playbook to add configmaps and secrets  https://review.openstack.org/44773320:03
*** rwallner has quit IRC20:03
*** rwellum has quit IRC20:03
*** rwellum_ is now known as rwellum20:03
*** rwallner has joined #openstack-kolla20:04
*** kbaegis has joined #openstack-kolla20:05
inc0lmao check "conflicts with" https://review.openstack.org/#/c/448258/20:05
sbezverkinc0: could you comment on one point. ansible bases its actions on an inventory file, in kolla kubernetes to have inventory file is against the nature of kubernetes cluster where you have no knowledge about your inventory. how it is going to work?20:06
inc0sbezverk: well, it *may* use inventory file20:06
inc0https://review.openstack.org/#/c/448258/1/ansible/site.yml20:06
sbezverkinc0: Ah so inventory is not a mandatory?20:07
inc0hosts: localhost connection:local20:07
sbezverkinc0: got it20:07
inc0you can specify this in play20:07
inc0and in our case, that's exactly what we're going to do20:07
*** rwallner has quit IRC20:08
sbezverkinc0: are you going to clean up this ps, it seems it has bunch of unrelated staff, things like reconfig and etc?20:10
inc0sbezverk: sure20:10
sbezverkinc0: cause now it looks scary ;)20:10
*** MasterOfBugs has joined #openstack-kolla20:11
sdakedims is go supported by the infra team currently?20:13
sdakedims I ask as your leading the go effort :)20:13
dimssdake : yes20:13
sdakeTIL :)20:13
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla-kubernetes master: WIP Move config generation to kolla-k8s  https://review.openstack.org/44825820:14
inc0sbezverk: ^ how about now?20:14
*** pbourke has quit IRC20:16
sbezverkinc0: man that looks way more human :)20:18
sdakeinc0 i don't htink we nbeed most of the library code20:18
inc0sdake: slowly20:18
sdakeand also need the templates for more then neutron20:18
inc0I think w ecan thin it up more20:18
inc0but I want to get it to work and test first20:18
inc0obviously...20:19
inc0I want to do one service at the time tho as it's easier to handle20:19
sdakewfm20:19
*** tonanhngo has joined #openstack-kolla20:19
sbezverkinc0: probably if you get config generation for one service as PoC then the rest of services should be easy to add20:19
inc0that's the plan20:19
inc0and since it's kinda tradition in kk8s20:20
sbezverkinc0: great20:20
inc0service is going to be neutron20:20
*** tonanhngo has quit IRC20:20
*** jemcevoy has quit IRC20:20
sbezverkinc0: cool, you can drop openvswitch things as they are not considerd under nuetron20:21
mnaseris kolla-ansible and kolla-k8s maintaining their own subtrees of ansible code?20:22
sdakeinc0 i think a path forward is your work -> generate helm charts for the cnofigmaps20:22
*** rhallisey has quit IRC20:22
inc0mnaser: not yet, but that's poc I'm working on now20:22
sdakemnaser only genconfig - not full workflow20:22
mnaseroh okay20:22
inc0for k8s tho it will be *MUCH* simpler one20:22
sdakemnaser and idea is to eject ansible/genconfig when we find a suitable replacement20:22
inc0we'll see about helm:P20:23
openstackgerritBertrand Lallau proposed openstack/kolla-ansible master: Use service_name variable when define in neutron role  https://review.openstack.org/44826920:23
*** lamt has quit IRC20:23
inc0I'd like to have discussion20:23
mnaserwhats genconfig20:23
inc0if I get this to work20:23
inc0mnaser: kolla-ansible genconfig20:23
inc0so "just render me a damn config files"20:23
sdakemnaser that is the thing that sets the default configuration in /etc/kolla for all of the services20:23
mnaserah okay okay20:23
mnasergotch20:23
mnaser+a20:23
sdakemnaser e.g. kolla-ansible genconfig will write out /etc/kolla/nova.conf20:23
mnaseri never used that feature but20:24
inc0that's how we create config files in kk8s today20:24
mnaserthat would have been nice to know20:24
mnaserwhen we were doing our migration :P20:24
mnaserTIL20:24
sdakemnaser its not really a feature we want t o keep in kolla-ansible ;)20:24
sdakealthough we are probably stuck with it if operators use it20:24
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla-kubernetes master: WIP Move config generation to kolla-k8s  https://review.openstack.org/44825820:24
inc0it's a good feature20:25
inc0if you want to ship hand crafted configs20:25
*** jemcevoy has joined #openstack-kolla20:25
inc0to carefully compare with what's going to happen once you run k-ans20:25
sdakeinc0 at some point you want to integrate with clouds.yaml20:26
sdakeinc0 what that integration means is up to you :)20:27
*** crushil has quit IRC20:27
inc0well, it's different tho (if we're talking of clouds.yaml as replacement of openrc)20:27
sdakesbezverk can explain the purpose of cloud.yaml much more eloquently then I can - I barely understand it - however its an override mechanism for config - not a  replacement for openrc20:28
inc0sbezverk: please...20:28
*** lamt has joined #openstack-kolla20:28
*** lamt has quit IRC20:29
sbezverkinc0: cloud yaml allows to describe your environment completely in terms of settings/values etc20:30
mnaserhttps://github.com/openstack/os-client-config/blob/master/README.rst20:30
sbezverkbasically once you build your cloud yaml, rebuilding the environment becomes a very easy task20:30
mnaserif you're up for a fun read20:30
mnaserunless we're talking20:31
mnaserabout another clouds.yaml..20:31
inc0sbezverk: cloud.yaml == globals.yaml for us20:31
sbezverkinc0: yes20:31
inc0want to change name from globals to clouds?:P20:31
inc0because we can do it:P20:31
inc0mnaser: I know of clouds.yaml as in "my endpoints are there, and my passwords"20:32
sbezverkinc0: not sure, cloud yaml is PERSONAL configuration20:32
inc0aka brave new openrc20:32
mnasersame inc020:32
sbezverkglobal.yaml is just a list of default variables20:32
mnaserok so20:32
mnaseri see where sbezverk  is coming from and20:32
mnaserwe did something simila20:32
inc0sbezverk: negative, globals are personal, defaults are in ansible dir20:32
*** dave-mccowan has joined #openstack-kolla20:32
mnaserwe maintain per-env folder that contains everything that /etc/kolla would contain for a specific env20:32
sbezverkinc0: like for me to rebuild kolla-kube takes very little effort20:33
mnaserand we run tools/run-kolla.sh <envname> deploy20:33
sbezverkas my cloud.yaml is exactly what I need it to be20:33
inc0well...again, you're talking about globals.yaml in kolla-ansible nomenclature;)20:33
mnaserwhich wraps around kolla-ansible and reconfigures the config dir to a local dir20:33
inc0mnaser: so if you run kolla-ansible20:34
inc0it will output ansible-playbook ....20:34
inc0(pretty long command)20:34
inc0as first line20:34
inc0kolla-ansible only builds this command20:34
mnaseroh yeah20:34
mnaserbut i figured its safer to keep kolla-ansible in there20:34
inc0so you can just wrap this one20:34
mnaserjust in case some things change20:34
inc0well I don't think we'll ever change this behavior20:35
inc0I wouldn't be ok with it20:35
mnaser(hell at first we integrated it by including the roles in role_path)20:35
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP Adding virtualbmc to the gate  https://review.openstack.org/44155020:35
inc0as kolla-ansible is just glorified alias20:35
mnaserso our very own site.yml would run the roles/include the site.yml from kolla20:35
mnaserbut that was starting to push some boundaries lo20:35
inc0yeah, site.yaml might change20:35
inc0kolla-ansible - unlikely20:35
inc0ansible-playbook allows you much more runtime modifications than kolla-ansible uses20:36
inc0and we want to keep this20:36
*** shashank_t_ has quit IRC20:37
sbezverkinc0: hmm I thought helm/all_values.yaml is more like global.yaml or maybe it is kind of mixture of both..20:37
*** tonanhngo has joined #openstack-kolla20:37
*** shashank_t_ has joined #openstack-kolla20:37
*** tonanhngo has quit IRC20:38
inc0helm/all_values would be equivalent to this: https://github.com/openstack/kolla-ansible/blob/master/ansible/group_vars/all.yml20:38
*** shashank_t_ has quit IRC20:38
inc0*this* is defaults file20:38
*** shashank_t_ has joined #openstack-kolla20:39
*** manheim has quit IRC20:39
sbezverkinc0: right20:39
*** shashank_t_ has quit IRC20:43
inc0sbezverk: you sir are wrong;) kubectl works with folder all righnt20:45
inc0if you run kubectl create -f /path/to/dir20:45
*** jemcevoy has quit IRC20:45
inc0it will try to create everything in dir that ends with .yaml or .json20:45
inc0it's neat20:45
sbezverkinc0: you do not want to combine them in a single yaml file?20:46
inc0no need imho20:47
sbezverkinc0: cause then you will need to loop through them20:47
inc0no20:47
inc0I'm saying20:47
inc0try to create multiple files in same dir20:47
inc0and run kubectl create on whole dir20:47
*** klindgren has joined #openstack-kolla20:48
inc0it will pick up all the files on it's own20:48
*** jemcevoy has joined #openstack-kolla20:48
sbezverkinc0: I have never seen kubectl capable of creating object for all files in a directory20:48
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla-kubernetes master: WIP: Playbook to add configmaps and secrets  https://review.openstack.org/44773320:49
inc0sbezverk: well...it is capable:)20:49
inc0check it out!20:49
sbezverk+ kubectl create --namespace=kolla -f /tmp/kolla-kubernetes/configmaps20:50
sbezverk2017-03-21 20:16:27.061001 | error: You must provide one or more resources by argument or filename.20:50
inc0ahh it fails on centos for different reason20:50
inc0which I just fixed (I hope)20:50
sbezverkinc0: if it is working, great20:50
inc0in ubuntu it works, in centos I have to install libselinux-python20:50
inc0which I just did20:51
inc0;_)20:51
*** athomas has joined #openstack-kolla21:05
*** shashank_t_ has joined #openstack-kolla21:07
*** crushil has joined #openstack-kolla21:09
*** lamt has joined #openstack-kolla21:10
*** shashank_t_ has quit IRC21:11
*** kbaegis has quit IRC21:17
*** kbaegis has joined #openstack-kolla21:17
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla-kubernetes master: WIP: Playbook to add configmaps and secrets  https://review.openstack.org/44773321:20
*** david-lyle has joined #openstack-kolla21:23
*** jtriley has quit IRC21:25
*** tonanhngo has joined #openstack-kolla21:25
*** jrobinson has joined #openstack-kolla21:25
*** tonanhngo has quit IRC21:27
*** Pavo has joined #openstack-kolla21:27
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla-kubernetes master: WIP Move config generation to kolla-k8s  https://review.openstack.org/44825821:32
*** shashank_t_ has joined #openstack-kolla21:33
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP Adding virtualbmc to the gate  https://review.openstack.org/44155021:34
*** Jeffrey4l_ has quit IRC21:35
*** crushil has quit IRC21:36
inc0kfox1111: around?21:37
openstackgerritBertrand Lallau proposed openstack/kolla-ansible master: Use service_name variable when defined in neutron role  https://review.openstack.org/44826921:38
*** Pavo has quit IRC21:40
kfox1111inc0: hi21:41
inc0I need halp:(21:41
kfox1111back21:41
kfox1111whats up?21:41
inc0https://review.openstack.org/#/c/447733/7/tools/setup_gate_common.sh looking at gate logs, line 23 doesn't seem to be run21:41
kfox1111inc0: you looking at the isci or ceph gate logs?21:43
inc0http://logs.openstack.org/33/447733/6/check/gate-kolla-kubernetes-deploy-centos-binary-2-ceph-nv/6695c7b/console.html#_2017-03-21_21_00_48_519453 <- I'm trying to debug thios21:43
kfox1111we're disabling selinux too.21:43
kfox1111grep for sudo yum install -y libselinux-python21:45
kfox1111http://logs.openstack.org/33/447733/6/check/gate-kolla-kubernetes-deploy-centos-binary-2-ceph-nv/6695c7b/console.html#_2017-03-21_20_50_23_86160321:45
kfox1111says its already installed.21:45
kfox1111maybe ansible's trying to talk to selinux even though its disabled?21:45
inc0omg21:45
inc0dmsimard: arouind by any chance?21:45
kfox1111sbezverk: around?21:46
kfox1111still need me to look at something?21:46
inc0kfox1111: also check this one out https://review.openstack.org/#/c/448258/21:46
kfox1111inc0: context for the conversation with jascott1 was for library helm charts. parent needs to win. totally seperate thing from the global api, in which case the more specific wins over the more generic.21:47
*** Jeffrey4l_ has joined #openstack-kolla21:48
*** haplo37 has quit IRC21:52
dmsimardinc0: on my phone right now, sup?21:53
inc0dmsimard: seen this in gates? http://logs.openstack.org/33/447733/6/check/gate-kolla-kubernetes-deploy-centos-binary-2-ceph-nv/6695c7b/console.html#_2017-03-21_21_00_48_51945321:54
*** lamt has quit IRC21:54
sbezverkkfox1111: hey21:54
dmsimardDoesn't ring me a bell but I've seen that problem before21:55
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla-kubernetes master: WIP: Playbook to add configmaps and secrets  https://review.openstack.org/44773321:55
dmsimardinc0: https://dmsimard.com/2016/01/08/selinux-python-virtualenv-chroot-and-ansible-dont-play-nice/21:55
sbezverkkfox1111: yeah if you could look at the link I posted  http://logs.openstack.org/50/441550/102/experimental/gate-kolla-kubernetes-deploy-ubuntu-source-3-ironic-nv/4bdd216/console.html#_2017-03-21_19_47_33_85445921:55
inc0yeah, I saw this, hence my asking...problem is I don't think this uses virtualenv21:56
inc0or chroot for that matter21:56
dmsimardtl;dr certain tasks require selinux bindings even on the control host because some operations happen there21:56
*** fguillot has quit IRC21:56
inc0yeah this play runs on local21:56
*** lamt has joined #openstack-kolla21:56
sbezverkkfox1111: ubuntu uis misbehaving when centos is perfectly ok..21:56
dmsimardinc0: doesn't sound like something that'd happen all of a sudden21:57
dmsimardThere's usually an ansible fact for python interpreter21:57
inc0yeah it's strange especially that kolla-ansible works, ansible-playbook doesnt21:58
inc0and kolla ansible is just wrapper around ansible playbook command21:58
inc0hmm that might be it21:58
inc0I disabled fact gathering21:58
dmsimardIs there an ara report in there ? Reading raw ansible output hurts my eyes now :P21:59
*** matrohon has quit IRC22:00
inc0lol22:00
inc0nope, no ara22:00
*** haplo37 has joined #openstack-kolla22:01
*** shashank_t_ has quit IRC22:09
*** vhosakot has quit IRC22:09
*** manheim has joined #openstack-kolla22:10
*** tonanhngo has joined #openstack-kolla22:12
*** tonanhngo has quit IRC22:13
*** athomas has quit IRC22:16
*** crushil has joined #openstack-kolla22:18
*** foliouno has quit IRC22:19
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla-kubernetes master: WIP Move config generation to kolla-k8s  https://review.openstack.org/44825822:21
*** klindgren has quit IRC22:22
*** Margin has joined #openstack-kolla22:26
*** Margin has quit IRC22:26
*** Margin has joined #openstack-kolla22:27
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla-kubernetes master: WIP: Playbook to add configmaps and secrets  https://review.openstack.org/44773322:27
*** lamt has quit IRC22:29
kfox1111sbezverk: so, the issue is:  qemu-system-x86_64: could not acquire pid file: Permission denied?22:31
kfox1111http://logs.openstack.org/50/441550/102/experimental/gate-kolla-kubernetes-deploy-ubuntu-source-3-ironic-nv/4bdd216/logs/libvirt-vm-logs.txt looks ok I think.22:33
sbezverkkfox1111: it looks like.. funny thing I have not seen similar error with newton22:36
kfox1111sbezverk: maybe there is an el7 specific feature in the xml?22:37
kfox1111or the feature uses a file location different between ubuntu and el722:37
sbezverkkfox1111: not sure, I mean I used only generic settings22:37
kfox1111did it get past this point in ubuntu before you added the bios file thingy?22:37
sbezverkkfox1111: nope I started seeing as soon as I switch to ocata22:38
sbezverkbtw check this http://logs.openstack.org/50/441550/104/experimental/gate-kolla-kubernetes-deploy-centos-source-3-ironic-nv/9940aa3/console.html#_2017-03-21_22_10_11_24763322:38
sbezverknow I see where VM got stuck22:38
kfox1111nice. :)22:38
kfox1111can you copy that into $logs/...  on failure?22:39
kfox1111only other thing I can think of on the ubuntu side is to setup some strace's and see if you can determine the file it cant open. :/22:40
sbezverkkfox1111: not sure because it gets kept in the container, if the container is still running when failure occur, then yes22:41
sbezverkkfox1111: ok I will look into ubuntu after centos is operational22:41
*** crushil has quit IRC22:42
kfox1111sbezverk: I think you can just move the command to cat the file to the gate_log script and if the containers still running, it will capture it. the script does a lot of things like that.22:43
kfox1111k.22:44
*** tonanhngo has joined #openstack-kolla22:44
*** adrian_otto has quit IRC22:44
*** tonanhngo has quit IRC22:45
sbezverkkfox1111: cool will add it.22:45
*** fooliouno has joined #openstack-kolla22:56
*** david-lyle has quit IRC22:57
*** cloudkiller has joined #openstack-kolla23:00
*** cloudkiller has quit IRC23:00
*** sayantani01 has quit IRC23:02
*** sayantani01 has joined #openstack-kolla23:03
*** sayantani01 has quit IRC23:03
*** cloudkiller has joined #openstack-kolla23:04
*** sayantani01 has joined #openstack-kolla23:04
*** eglute_s has joined #openstack-kolla23:04
*** sayantani01 has quit IRC23:06
*** harlowja has quit IRC23:06
*** sayantani01 has joined #openstack-kolla23:06
*** eglute_s has quit IRC23:07
*** cloudkiller has quit IRC23:07
*** goldyfruit has quit IRC23:09
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP Adding virtualbmc to the gate  https://review.openstack.org/44155023:16
*** Margin has quit IRC23:17
*** sayantani01 has quit IRC23:20
*** sayantani01 has joined #openstack-kolla23:21
*** rwsu has joined #openstack-kolla23:22
*** gfhellma has quit IRC23:26
spsuryamorning all23:29
*** jtriley has joined #openstack-kolla23:32
*** cloudkiller has joined #openstack-kolla23:34
*** eglute_s has joined #openstack-kolla23:34
*** eglute_s has quit IRC23:46
*** cloudkiller has quit IRC23:46
*** cloudkiller has joined #openstack-kolla23:47
*** eglute_s has joined #openstack-kolla23:47
masberhi23:48
*** eglute has quit IRC23:50
*** cloudnull has quit IRC23:50
*** cloudkiller is now known as cloudnull23:50
*** manheim has quit IRC23:53
masberI am testing kolla-ansible installation, I tried different interfaces configuration but my vms doesn't seems like they have internet access. According to the documentation neutron_external_interface is the one connected to br-ex so I am assuming the external traffic from/to my vms will go through that interface is that right?23:56

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!