Thursday, 2017-01-26

sbezverkkfox1111: by some reason init container is missing mount!?!?!00:00
kfox1111trunk or on a ps?00:00
*** sayantani01 has quit IRC00:00
sbezverkkfox1111: so the extra mount is not working :(00:00
*** sdake has quit IRC00:01
kfox1111oh... we're not using config.json at all here.... that makes it easier.00:02
*** lamt has quit IRC00:02
sbezverkkfox1111: it is crazy http://paste.openstack.org/show/596526/00:03
kfox1111sbezverk: oh. hyou have deps twice.00:04
kfox1111you don't need to define value dependencies for that socket.00:04
kfox1111as there always is an explicit one.00:04
sbezverkkfox1111: that part I knew00:05
sbezverkbecause we needed one when deployed as a microservice00:06
kfox1111yeah.00:06
sbezverkand then for consistency with the rest we added00:06
kfox1111so should be removed from the service package.00:06
sbezverkanother it shold not be a problem00:06
sbezverkthe problem is why it does not render required volume00:06
kfox1111it doesn't have mounts though, so it won't work.00:06
sbezverkexactly00:06
kfox1111your saying the extra dep container should hve mounts too?00:07
sbezverkabsolutely it should00:07
kfox1111ah. ok. gotcha. sec...00:07
sbezverkotherwise it does not have access to socket00:07
kfox1111oh...00:09
kfox1111I see why.00:09
kfox1111helm/kolla-common/templates/_common_dependency_container.yaml00:09
kfox1111definitions of common_dependency_container_single and common_dependency_container_multi00:10
kfox1111they don't add extraDepVolumeMounts to the child call if they exist.00:10
sbezverkkfox1111: that explains it00:11
kfox1111each needs a:00:12
kfox1111{{- if hasKey .Values "extraDepVolumeMounts" }}00:12
kfox1111{{- $_ := set $c1 "extraDepVolumeMounts" .extraDepVolumeMounts }}00:12
kfox1111{{- end }}00:12
kfox1111the more confusing question, is how did this work in the gate ever.00:12
kfox1111or did you add the dep to your own values.yaml?00:12
kfox1111its in the openvswitch valules file.00:13
kfox1111that is very unsettling.... :/00:13
kfox1111this should be broken I think...00:13
sbezverkkfox1111: one sec00:14
sbezverkboth macros use common_dependency_container_common00:14
sbezverkwhere there is a check for extra mount00:14
sbezverkkfox1111: Am I missing something?00:15
kfox1111yeah. but common_dependency_container_multi and common_dependency_container_single builds the '.' environment for that function.00:15
kfox1111those 3 lines above need to be added to both, to pass it on through.00:15
kfox1111oh. wait...00:15
kfox1111I'm totally missreading it.00:16
kfox1111its building args to kolla_val_get_raw, not common_dependency_container_common00:16
kfox1111common_dependency_container_common is getting . from the multi/single macros.00:16
kfox1111disreguard what I said... :/00:17
kfox1111ok. lets walk through it again...00:17
kfox1111helm/microservice/openvswitch-vswitchd-daemonset/templates/openvswitch-vswitchd-daemonset.yaml defines openvswitch_vswitchd_extra_dep_volume_mounts00:17
kfox1111sets {{- $extraDepVolumeMounts := "openvswitch_vswitchd_extra_dep_volume_mounts" }}00:17
kfox1111adds $extraDepVolumeMounts  as "extraDepVolumeMounts" to $env00:18
sbezverkright00:18
kfox1111then calls common_dependency_container_multi with $env.00:18
*** huzhengchuan has joined #openstack-kolla00:18
kfox1111which passes it directly to common_dependency_container_common00:18
kfox1111then checks for {{- if .extraDepVolumeMounts }}00:19
kfox1111I see no with's there in that macro.00:20
sbezverkyou see it does not even render volumes:00:20
sbezverkno that if is broken somehow00:20
kfox1111yeah. its acting like .extraDepVolumeMounts is false.00:20
kfox1111all the variable names match up...00:22
kfox1111this should be working like common_api_apache deployment.00:23
kfox1111which is working.00:23
sbezverkany chance we lose some vars when call include "common_dependency_container_common" .00:23
kfox1111its also working in the gate.00:23
sbezverkI mean '.' instead of $env00:24
kfox1111. comes from the parent,00:24
kfox1111which was called with $env.00:24
sbezverkyou see {{- if $localVals.kubernetes_entrypoint }}00:24
sbezverkworks fine00:25
kfox1111I don't nkow how you can be seeing that. :/00:26
sbezverkkfox1111: if it was not working then extra dependency container would not show up00:27
kfox1111I don't follow.00:28
kfox1111trunk openvswitch service package has the dep in there the same as yours.00:28
*** eaguilar has joined #openstack-kolla00:28
kfox1111so should have both init containers.00:28
kfox1111somehow its working though.00:28
kfox1111I think...00:29
kfox1111yeah. cause https://review.openstack.org/#/c/424921/ passed after that merged...00:29
sbezverkkfox1111: I should be able to see what is stored in '.'00:31
sbezverkrebuilding microservices and service00:31
kfox1111yeah. can just dump it out.00:31
kfox1111foo: {{ . }}00:31
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: Add dep for neutron-openvswitch-agent on a working openvswitch.  https://review.openstack.org/42537600:34
kfox1111arg.. committed a while ago but didn't review. :/00:34
kfox1111there's 10 minutes of confuse debugging down the drain. :/00:34
*** inc0 has quit IRC00:35
kfox1111yay. trunk builds back to almost sort of kind of working: http://logs.openstack.org/50/418550/58/check/gate-kolla-kubernetes-deploy-centos-binary-t-ceph-multi-nv/094a42c/console.html00:35
kfox1111just cinder left.00:35
*** duonghq has joined #openstack-kolla00:38
duonghqmorning guys00:38
kfox1111duonghq: morning.00:39
sbezverkduonghq: morning00:39
kfox1111sbezverk: did you build those by hand, or by tools/helm_build_all.sh?00:39
duonghqhi kfox1111 sbezverk00:39
kfox1111maybe you got an earlier version somehow?00:39
kfox1111or... if you have the package that's doing that,00:39
kfox1111could you extract the tarball and look at the deps template?00:39
sbezverkkfox1111: yes and what is interesting when I dump from microservice dry run I do see extravolume00:40
kfox1111waa?00:40
sbezverkwhen I do the same from service level it does not00:40
kfox1111but, when you load it, then get daemonset that you put in pastebin, its not there?00:40
*** portdirect is now known as portdirect_travl00:40
sbezverkkfox1111: I explicitely specify kubernetes_entrypoint in microservice00:40
kfox1111with dryrun?00:40
sbezverkkfox1111: http://paste.openstack.org/show/596529/00:41
sbezverkthis is for microservice00:42
*** sayantani01 has joined #openstack-kolla00:43
sbezverkkfox1111: here is for service http://paste.openstack.org/show/596530/00:43
openstackgerritsayantani proposed openstack/kolla-ansible: Fix RabbitMQ server error in bifrost  https://review.openstack.org/42438400:45
kfox1111this feels like a bug in helm. :/00:45
openstackgerritsayantani proposed openstack/kolla-ansible: Fix MySQL DB error in bifrost container  https://review.openstack.org/42439100:45
openstackgerritsayantani proposed openstack/kolla-ansible: Update Bifrost playbooks  https://review.openstack.org/42543500:46
kfox1111uhhh........00:48
*** tovin07 has joined #openstack-kolla00:49
*** tovin07_ has joined #openstack-kolla00:49
kfox1111oh...00:50
kfox1111this is hard to parse since there is more then one template in the logs. :/00:51
*** dave-mcc_ has joined #openstack-kolla00:51
*** krtaylor has joined #openstack-kolla00:51
kfox1111ok. ran it a few times.00:52
kfox1111I'm seeing the volume mounts consistently on both service package and microservice package.00:53
kfox1111I was sometimes loking at the ovsdb one and that was throwing me for a bit.00:53
*** dave-mccowan has quit IRC00:53
kfox1111sbezverk: do you staill have access to that service package?00:54
kfox1111can you post it somehwere?00:54
kfox1111wow... all the other cores are gone. :/00:55
sbezverkkfox1111: really sorry, it was my fault :(00:55
sbezverkit turned out there were two helm repo00:55
sbezverkso the service package was getting built from older repo00:56
sbezverkafter cleaning everything up it started working00:57
*** cuongnv has joined #openstack-kolla00:57
kfox1111ok. cool.00:58
kfox1111tools/helm_build_all.sh. :)00:58
kfox1111glad you found the issue though.00:59
kfox1111oh darn...01:00
duonghqdoes the portdirect workaround of docker 1.13.0 issue work for you?01:01
kfox1111that appctl command is being way to clever...01:01
duonghqthe tiler problem01:01
kfox1111its looking at the pid, deciding its not in the pid namespace, then killing the file. :/01:01
openstackgerritDuong Ha-Quang proposed openstack/kolla-ansible: Specify 'become' for only neccesary tasks (default roles)  https://review.openstack.org/39868401:01
openstackgerritDuong Ha-Quang proposed openstack/kolla-ansible: Specify 'become' for only neccesary tasks (all other roles)  https://review.openstack.org/39868501:02
kfox1111duonghq: who is that question for?01:02
sbezverklater gents, talk to you tomorrow..01:02
portdirect_travlWorks for me, but I'm definitely not trustworthy on that front :)01:03
duonghqkfox1111, you and sbezverk01:03
duonghqsorry, I forgot mention01:03
kfox1111duonghq: I dont have the issue as I avoid docker 1.13 alltogether. I trust redhat's docker more.01:03
*** jpstnh has quit IRC01:03
portdirect_travlThink technopolis has been pointing others people to that image as well :) and :/01:03
duonghqkfox1111, ok01:03
duonghqportdirect_travl, your image still crashbackloop in my halcyon setup, cannot figure out why01:04
kfox1111portdirect_travl: can you please review https://review.openstack.org/#/c/424921/ real quick.01:04
kfox1111and if you have another minute, https://review.openstack.org/#/c/424368/ .01:05
portdirect_travlduonghq: is your networking up? Most of the time I see tiller misbehaving DNS is down01:05
kfox1111both are parts to getting the gate stable again.01:05
portdirect_travlOn phone kfox, can you give me the rundown01:06
portdirect_travl(Tiny screens and gerrit...)01:06
*** bjolo_ has joined #openstack-kolla01:06
kfox1111portdirect_travl: one builds cache tarballs that we will be able to download in the gate instead of from the docker hub. for kolla nad kubernetes/helm containers.01:06
kfox1111so more stable access to them.01:06
duonghqportdirect_travl, http://paste.openstack.org/show/596532/01:06
kfox1111the other just adds a bit more error checking.01:06
*** huikang has joined #openstack-kolla01:06
portdirect_travlRoger, will check as best I can01:07
kfox1111thx.01:07
kfox1111the first one is more important, but hte second is a very easy review.01:08
*** sacharya has joined #openstack-kolla01:08
portdirect_travlDuonghq, that's your k8s dashboard in the crashloop not tiller01:08
duonghqportdirect_travl, sorry, misreading this01:08
portdirect_travlThough that's not a good sign01:08
duonghqportdirect_travl, moment, I'm searching your workaround01:09
portdirect_travlkfox1111: could you see if you could help duonghq ?01:09
kfox1111yeah. I have a little bit more time.01:09
kfox1111whats kubectl get pods --all-namespaces show?01:10
portdirect_travlnice multitasking on a 5 inch screen ain't fun :)01:11
kfox1111I get that. :)01:11
duonghqkfox1111,  http://paste.openstack.org/show/596532/  <- this is the output01:11
kfox1111k.01:12
kfox1111and: kubectl describe pod tiller-deploy-3299276078-nm5q7 --namespace kube-system01:12
duonghqkfox1111, I applied portdirect_travl image, and get follow output:01:13
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: Add dep for neutron-openvswitch-agent on a working openvswitch.  https://review.openstack.org/42537601:14
duonghqhttp://paste.openstack.org/show/596533/ http://paste.openstack.org/show/596534/01:14
portdirect_travlkfox1111: am i'm being mental or does this do the opposite of what the commit message say? https://review.openstack.org/#/c/424368/01:15
kfox1111hmm...01:15
kfox1111portdirect_travl: its a bit confusing to write about, because bash does flags backwards.01:17
portdirect_travlI'm being mental01:17
kfox1111 -e adds the flag, +e removes it. :/01:17
portdirect_travlyes01:17
portdirect_travloh well least i embarrassed myself on irc and not gerrit :)01:17
kfox1111duonghq: thats a strange output. it looks like it pulled the image ok, but then is then shooting it.01:18
kfox1111portdirect_travl: no worries. That trips me up alot too.01:18
portdirect_travlthat dashboard crashloop makes me think that the pods cannot talk to the api server01:18
duonghqkfox1111, yes01:19
kfox1111yeah.01:19
kfox1111how about: kubectl get pods --all-namespaces -o wide01:19
*** zhubingbing has joined #openstack-kolla01:19
kfox1111lets see if there is a pattern to the nodes landed on.01:19
*** dave-mccowan has joined #openstack-kolla01:20
duonghqkfox1111, http://paste.openstack.org/show/596535/01:20
kfox1111portdirect_travl: thanks for the reviews.01:20
kfox1111so, a slight pattern.01:21
kfox1111there are only 3 pods using non net=host.01:21
kfox1111of those, 1 is wokring, and on the head node it apears,01:21
kfox1111and 2 are not, and on a different node.01:22
kfox1111I would guess weave is not working properly.01:22
*** dave-mcc_ has quit IRC01:22
duonghqI saw weave on .14 is restart once01:22
kfox1111it restarted on 13 too.01:23
kfox1111that may be normal.01:23
kfox1111I know nothing about weave though.01:23
kfox1111been using canal.01:23
kfox1111portdirect_travl: what are you using?01:23
openstackgerritMerged openstack/kolla-kubernetes: Capture lists of containers by type and cache containers.  https://review.openstack.org/42492101:23
kfox1111(for testing)01:23
duonghqbut there is not any non net=host pod on node 13, I guess that if any, this'll get trouble too, not sure how to reschedule pod to run on other node01:24
portdirect_travlcalico01:25
kfox1111portdirect_travl: k. thx.01:25
kfox1111duonghq: maybe try calico instead of weave?01:25
kfox1111thought that may require burning down the cluster to switch.01:26
portdirect_travlused to hate it - then one day (about 9 months ago) it started working and I've never thought about it since really01:26
duonghqkfox1111, sure01:26
duonghqportdirect_travl, which calico version are you using?01:26
portdirect_travlv2.001:26
duonghqack01:26
portdirect_travlif you are running halcyon - you can chance one line in the playbook to use it01:27
duonghqportdirect_travl, saw it01:27
kfox1111portdirect_travl: oh. pure calico? not the canal variant?01:27
kfox1111any thoughts about which one to use?01:27
portdirect_travlso canals just calico's security groups with flannel01:28
*** Syffs has quit IRC01:29
portdirect_travlso its the choice between tunnels and routing really01:29
duonghqwith my network, take down and bring up again the cluster take awhile to be done01:29
portdirect_travlif you can I'd go for plain calico01:29
*** bpattewar has quit IRC01:29
portdirect_travlbut untill you have a lot of nodes i dont think it makes much of a difference01:30
kfox1111k.01:30
openstackgerritDuong Ha-Quang proposed openstack/kolla: Apply neutron database migration  https://review.openstack.org/40784001:31
*** sdake has joined #openstack-kolla01:33
kfox1111there we go...01:34
kfox1111maybe a fix for the openvswitch startup thing finally.01:34
*** sdake_ has joined #openstack-kolla01:35
*** sdake has quit IRC01:38
*** hfu has joined #openstack-kolla01:40
*** huikang has quit IRC01:41
*** huikang has joined #openstack-kolla01:41
openstackgerritDuong Ha-Quang proposed openstack/kolla-ansible: Apply neutron database migration  https://review.openstack.org/40792201:42
*** sdake_ has quit IRC01:43
kfox1111gota head out. l8r all.01:44
duonghqkfox1111, cya01:44
*** eaguilar has quit IRC01:46
*** huikang has quit IRC01:47
*** sdake has joined #openstack-kolla01:49
duonghqsdake, sup01:49
*** Jeffrey4l_ has quit IRC01:49
*** Jeffrey4l has joined #openstack-kolla01:49
*** sdake_ has joined #openstack-kolla01:50
*** erlon has quit IRC01:50
*** sdake has quit IRC01:54
*** zhubingbing has quit IRC01:54
*** dave-mcc_ has joined #openstack-kolla01:57
*** dave-mccowan has quit IRC01:59
*** zhubingbing has joined #openstack-kolla02:02
*** sacharya_ has joined #openstack-kolla02:07
*** sacharya has quit IRC02:08
*** sdake_ has quit IRC02:09
*** huzhengchuan has quit IRC02:09
*** unicell has quit IRC02:10
*** sdake has joined #openstack-kolla02:11
*** rwellum has quit IRC02:17
*** Mech422 has quit IRC02:42
*** Jeffrey4l has quit IRC02:45
*** sacharya_ has quit IRC02:50
*** sacharya has joined #openstack-kolla02:50
*** sayantani01 has quit IRC02:50
*** markmcclain has quit IRC02:53
*** sdake has quit IRC02:58
*** markmcclain has joined #openstack-kolla02:58
*** prameswar has joined #openstack-kolla03:03
*** lrensing has joined #openstack-kolla03:09
*** qiliang27 has quit IRC03:13
*** lamt has joined #openstack-kolla03:17
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for kuryr  https://review.openstack.org/42416103:17
*** lrensing has quit IRC03:18
*** eaguilar has joined #openstack-kolla03:18
*** eaguilar has quit IRC03:19
*** eaguilar has joined #openstack-kolla03:20
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfigure action for cloudkitty  https://review.openstack.org/42251603:22
*** newmember has quit IRC03:23
*** rushil has quit IRC03:23
*** bjolo_ has quit IRC03:31
*** zhubingbing has quit IRC03:32
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for ironic  https://review.openstack.org/42390103:40
*** rushil has joined #openstack-kolla03:41
*** sdake has joined #openstack-kolla03:46
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfigure action for trove  https://review.openstack.org/42399004:04
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for ironic  https://review.openstack.org/42390104:07
*** sdake_ has joined #openstack-kolla04:08
*** sdake has quit IRC04:09
*** williamcaban has joined #openstack-kolla04:27
*** sdake has joined #openstack-kolla04:31
*** sdake_ has quit IRC04:32
*** dave-mcc_ has quit IRC04:32
*** eaguilar has quit IRC04:34
*** cristicalin has joined #openstack-kolla04:42
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for ironic  https://review.openstack.org/42390104:43
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for ironic  https://review.openstack.org/42390104:45
*** sdake has quit IRC04:49
*** mdnadeem has joined #openstack-kolla04:50
*** sayantani01 has joined #openstack-kolla04:57
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for influxdb  https://review.openstack.org/42415904:59
*** britthouser1 has joined #openstack-kolla05:03
*** britthouser has quit IRC05:03
*** msimonin1 has joined #openstack-kolla05:04
*** zhubingbing has joined #openstack-kolla05:05
*** msimonin has quit IRC05:06
*** cristicalin has quit IRC05:08
*** zhubingbing has quit IRC05:09
*** zhubingbing has joined #openstack-kolla05:10
*** zhubingbing has quit IRC05:12
*** wcb has joined #openstack-kolla05:12
*** sacharya has quit IRC05:14
*** williamcaban has quit IRC05:14
*** sacharya has joined #openstack-kolla05:14
*** sacharya has quit IRC05:15
*** sacharya has joined #openstack-kolla05:15
*** sacharya has quit IRC05:20
*** sdake has joined #openstack-kolla05:25
*** unicell has joined #openstack-kolla05:28
*** cristicalin has joined #openstack-kolla05:30
openstackgerritCuong Nguyen proposed openstack/kolla: Update elastic repo to https  https://review.openstack.org/42553905:32
*** mdnadeem has quit IRC05:36
openstackgerritcaoyuan proposed openstack/kolla-ansible: [WIP]Optimize reconfiguration for haproxy  https://review.openstack.org/42415605:38
*** srwilkers_ has quit IRC05:39
*** cristicalin has quit IRC05:40
*** sdake has quit IRC05:45
*** mdnadeem has joined #openstack-kolla05:48
*** unicell has quit IRC05:53
*** unicell has joined #openstack-kolla05:54
*** wcb has quit IRC06:00
*** mdnadeem has quit IRC06:02
*** mdnadeem has joined #openstack-kolla06:02
*** rushil has quit IRC06:04
*** sdake has joined #openstack-kolla06:06
*** sacharya has joined #openstack-kolla06:16
*** sdake has quit IRC06:16
*** sdake has joined #openstack-kolla06:16
*** sdake_ has joined #openstack-kolla06:21
*** sacharya has quit IRC06:22
*** sdake has quit IRC06:23
*** lamt has quit IRC06:27
*** sdake_ has quit IRC06:27
*** saneax has joined #openstack-kolla06:34
*** zioproto has joined #openstack-kolla06:35
*** zioproto has quit IRC06:41
*** saneax is now known as saneax-_-|AFK06:46
*** prameswar has quit IRC06:59
*** imcsk8 has quit IRC07:02
*** imcsk8 has joined #openstack-kolla07:04
*** jmccarthy has joined #openstack-kolla07:07
*** bmace_ has quit IRC07:07
*** bmace_ has joined #openstack-kolla07:07
*** sayantani01 has quit IRC07:15
*** jmccarthy has quit IRC07:15
openstackgerritCuong Nguyen proposed openstack/kolla: Update elastic repo to https  https://review.openstack.org/42553907:21
*** zhubingbing has joined #openstack-kolla07:32
*** msimonin1 has quit IRC07:41
*** duonghq has quit IRC07:51
*** hfu has quit IRC07:55
*** hfu has joined #openstack-kolla07:55
*** hfu has quit IRC07:55
*** hfu has joined #openstack-kolla07:56
*** hfu has quit IRC07:56
*** hfu has joined #openstack-kolla07:57
*** hfu has quit IRC07:57
*** hfu has joined #openstack-kolla07:57
*** hfu has quit IRC07:58
*** hfu has joined #openstack-kolla07:58
*** hfu has quit IRC07:59
*** tovin07 has quit IRC08:02
*** tovin07_ has quit IRC08:05
*** hieulq has quit IRC08:09
*** zioproto has joined #openstack-kolla08:12
*** zioproto1 has joined #openstack-kolla08:16
*** zioproto has quit IRC08:20
*** zioproto1 has quit IRC08:27
*** zioproto has joined #openstack-kolla08:30
*** zioproto has quit IRC08:31
*** zioproto has joined #openstack-kolla08:31
*** britthouser1 has quit IRC08:32
*** msimonin has joined #openstack-kolla08:33
*** britthouser has joined #openstack-kolla08:34
*** egonzalez has joined #openstack-kolla08:37
*** simon-AS559 has joined #openstack-kolla08:38
*** britthouser2 has joined #openstack-kolla08:44
*** britthouser has quit IRC08:44
*** gfidente has joined #openstack-kolla08:48
*** gfidente has quit IRC08:48
*** gfidente has joined #openstack-kolla08:48
*** unicell has quit IRC08:54
*** shardy has joined #openstack-kolla09:02
*** matrohon__ has joined #openstack-kolla09:03
*** pomac has joined #openstack-kolla09:12
*** athomas has joined #openstack-kolla09:21
*** sambetts|afk is now known as sambetts09:26
*** mgoddard has joined #openstack-kolla09:27
pomacAny comments on: https://bugs.launchpad.net/kolla-ansible/+bug/1659507 - I'm still trying to figure out why my gerrit account doesn't work09:33
openstackLaunchpad bug 1659507 in kolla-ansible "[RFC] push libvirt-guests to enable compute node vm resumption" [Undecided,New]09:33
pomacrotfl09:33
pomacehm..09:33
*** Serlex has joined #openstack-kolla09:35
*** shardy has quit IRC09:42
*** shardy has joined #openstack-kolla09:42
*** rmart04 has joined #openstack-kolla09:46
egonzalezpomac, have you changed your community profile to be "Foundation Member" instead of "Community Member"?09:46
pomacegonzalez: most likely not - how do you do that?09:47
pomacegonzalez: i registered a long time ago but never completed the process, so everything is in the same state as 'eons ago'09:48
egonzalezhttps://www.openstack.org/profile/09:48
pomacegonzalez: oh, I'm being signed in via launchpad in gerrit currently09:49
egonzalezyup, but also need to add you community profile as foundation, if not gerrit can't retrieve you info09:49
pomacah09:50
*** matrohon__ has quit IRC09:53
pomacegonzalez: i would prefer to be a community member, but since i don'09:54
pomact have anything today, maybe thats enough09:54
*** matrohon__ has joined #openstack-kolla09:55
pomacegonzalez: still can't store contact information...10:00
*** cuongnv has quit IRC10:01
egonzalezsigned https://review.openstack.org/#/settings/agreements ?10:01
egonzalezOpenStack Individual Contributor License Agreement10:01
pomacegonzalez: So, when i try to do that or update my "contact information" i get the same error10:01
pomac"Server Error10:02
pomacCannot store contact information"10:02
pomacegonzalez: i talked some with Khai Do yesterday and he couldn't see anything wrong at the time10:02
egonzalezis your primary email address in gerrit the same aas primary in openstack profile?10:03
pomacyes, but the launchpad one is still toating a old one - can't seem to get it removed10:04
pomacegonzalez: (it only shows up when i authenitcate, can't see it anywhere else on launchpad10:05
pomacheyyy10:06
pomacLogged out of launcpad, ubuntuone and god knows what else, the account looks better npw10:07
pomacnow10:07
egonzalezthis auth system with 3 differents accounts is madness for newcomers10:08
pomacegonzalez: still can't sign or set my address tough10:09
pomacegonzalez: (i'm much more used to being on the otherside of gerrit.... )10:09
*** simon-AS5591 has joined #openstack-kolla10:15
pomacegonzalez: (tried again using incognitomode - so nothing esle should be problematic)10:18
egonzalezpomac: worked?10:18
*** simon-AS559 has quit IRC10:18
pomacegonzalez: nope10:19
*** gema has quit IRC10:21
pomacegonzalez: i bet that there is a exception somewhere in the log that would explain this ;)10:21
pomaclunchtime, afk10:22
*** pbourke has quit IRC10:30
*** pbourke has joined #openstack-kolla10:30
*** zioproto has quit IRC10:30
*** zioproto has joined #openstack-kolla10:31
*** shardy has quit IRC10:51
*** shardy has joined #openstack-kolla10:52
*** eaguilar has joined #openstack-kolla11:12
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for influxdb  https://review.openstack.org/42415911:13
*** openstackgerrit has quit IRC11:17
*** matrohon__ has quit IRC11:29
*** shardy has quit IRC11:32
*** shardy has joined #openstack-kolla11:34
*** eaguilar has quit IRC11:37
*** eaguilar has joined #openstack-kolla11:49
*** openstackgerrit has joined #openstack-kolla11:59
openstackgerritMauricio Lima proposed openstack/kolla-ansible: Optimize reconfiguration for collectd  https://review.openstack.org/42413311:59
*** pc_m has quit IRC12:01
openstackgerritMerged openstack/kolla-ansible: Add chrony ansible role  https://review.openstack.org/39935512:03
*** pc_m has joined #openstack-kolla12:10
*** shardy is now known as shardy_lunch12:10
zioprotoegonzalez, how to I find the blueprint you mention here ? https://review.openstack.org/#/c/425263/12:11
zioprotoI was able to run a multimode deployment just hacking here and there some become: true12:11
egonzalezzioproto: https://blueprints.launchpad.net/kolla-ansible/+spec/ansible-specific-task-become12:14
zioprotothanks12:14
egonzalezrelated PS https://review.openstack.org/#/c/398682/512:14
zioprotoThere is anywhere described the architecture deployed with the Kolla multimode ?12:16
zioprotoI see the HA proxy is deployed on the network node12:16
zioprotobut if you have only one12:16
zioprotobasically there is only 1 ha proxy, right ?12:17
zioprotothere is no VRRP involved in this case i guess12:17
egonzalezyup, i'm using allinone with haproxy12:17
zioprotook, I have finished my first deployment12:17
*** eaguilar has quit IRC12:17
zioprotoso, we want to deploy Kolla by steps12:17
zioprotoas first step we would like to deploy just the mariadb part12:17
*** eaguilar has joined #openstack-kolla12:18
zioprotobut then we need to change our inventory to deploy somewhere ha proxy12:18
egonzalezmariadb will need haproxy12:18
zioprotoyes I see that12:18
zioprotohow much is a operator free to hack the inventory to change the deployment ?12:18
egonzalezas the database VIP is bind there12:18
zioprotowell, is not really a VIP, is the IP of HAProxy12:18
zioprotoit is not a VIP elected with VRRP if you have only 1 haproxy12:19
zioprotoI found confusing this name12:19
zioprotodo the Kolla playbook support to have more than 1 HA proxy ?12:19
zioprotoI will go to read that blueprint now12:19
egonzalezyou know, if have more than one will become a VIP ;)12:19
zioprotoright  :)12:19
zioprotoI think HA proxy should be added to this because you call sysctl in the playbook: https://review.openstack.org/#/c/398682/12:23
zioprotoshould I propose a patchset 6 on this review ?12:23
zioprotowhat is the IRC nick of Duong ?12:23
*** eaguilar has quit IRC12:25
*** erlon has joined #openstack-kolla12:27
egonzalezzioproto: is added https://review.openstack.org/#/c/398682/5/ansible/roles/haproxy/tasks/config.yml12:28
egonzalezbut no in the list at commit description12:28
zioprotoright12:28
zioprotoI will comment12:28
zioprotoon the review12:29
*** matrohon__ has joined #openstack-kolla12:31
egonzalezis duongh IIRC, but is not around here at this time. Use to be here in 3-4h before and later this time12:32
egonzalezanybody have an idea why this PS is not being tested by CI infra? https://review.openstack.org/#/c/405215/712:42
openstackgerritcaoyuan proposed openstack/kolla-ansible: [WIP]Optimize reconfiguration for tempest  https://review.openstack.org/42418812:44
*** lamt has joined #openstack-kolla12:49
*** lamt has quit IRC12:53
*** eaguilar has joined #openstack-kolla12:54
*** l4yerffeJ__ has joined #openstack-kolla12:56
*** l4yerffeJ_ has quit IRC12:56
openstackgerritEduardo Gonzalez proposed openstack/kolla-ansible: Implement Keystone zero-downtime upgrade  https://review.openstack.org/42544613:03
openstackgerritMauricio Lima proposed openstack/kolla-ansible: Optimize reconfiguration for aodh  https://review.openstack.org/42367013:03
mliimamorning guys13:04
*** shardy_lunch is now known as shardy13:04
openstackgerritEduardo Gonzalez proposed openstack/kolla-ansible: Implement Keystone zero-downtime upgrade  https://review.openstack.org/42544613:05
openstackgerritEduardo Gonzalez proposed openstack/kolla-ansible: Implement Keystone zero-downtime upgrade  https://review.openstack.org/42544613:11
*** eaguilar has quit IRC13:11
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfigure action for vmtp  https://review.openstack.org/42419413:22
*** rwellum has joined #openstack-kolla13:31
*** srwilkers has joined #openstack-kolla13:34
*** papacz has quit IRC13:34
*** Jeffrey4l has joined #openstack-kolla13:35
*** papacz has joined #openstack-kolla13:36
*** sdake has joined #openstack-kolla13:38
sdakemorning peeps13:39
egonzalezmorning sdake13:40
sdakesup egonzalez13:43
*** mgiles has joined #openstack-kolla13:45
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfigure action for vmtp  https://review.openstack.org/42419413:45
Jeffrey4lmorning.13:47
Jeffrey4legonzalez, could u review this again? https://review.openstack.org/41604613:47
mliimamorning sdake egonzalez Jeffrey4l13:47
egonzalezJeffrey4l: doing13:48
Jeffrey4lthanks13:48
openstackgerritJeffrey Zhang proposed openstack/kolla: Use install-pip macro in most source images  https://review.openstack.org/41386113:48
egonzalezsuch amount of quotes get me blind ;)13:49
sdakeJeffrey4l if I dont' see ya tomorrows on chinese new year, happy new year!!13:50
*** jheroux has joined #openstack-kolla13:51
Jeffrey4lthanks ;)13:51
*** zhubingbing has joined #openstack-kolla13:52
zhubingbinghey guys13:52
zhubingbinghttps://review.openstack.org/#/c/402224/13:52
zhubingbingcan u help me review it ?13:52
*** zhubingbing has quit IRC13:53
openstackgerritMerged openstack/kolla-ansible: Optimize reconfiguration for influxdb  https://review.openstack.org/42415913:54
*** goldyfruit has joined #openstack-kolla13:54
openstackgerritMerged openstack/kolla-kubernetes: Add dep for neutron-openvswitch-agent on a working openvswitch.  https://review.openstack.org/42537613:55
sdakezhenguo same to you re chinese new year :)13:57
*** rhallisey has joined #openstack-kolla14:02
*** zkun has joined #openstack-kolla14:04
openstackgerritzhang kun proposed openstack/kolla-ansible: Fix Bootstraping the mongodb replication set  https://review.openstack.org/42237714:04
*** zkun has quit IRC14:10
*** saneax-_-|AFK is now known as saneax14:11
srwilkersmorning14:12
*** shardy has quit IRC14:14
*** zhubingbing has joined #openstack-kolla14:14
*** shardy has joined #openstack-kolla14:15
*** haplo37_ has quit IRC14:15
openstackgerritMerged openstack/kolla-ansible: Optimize reconfiguration for tempest  https://review.openstack.org/42418814:16
*** hfu has joined #openstack-kolla14:17
*** haplo37_ has joined #openstack-kolla14:18
*** eaguilar has joined #openstack-kolla14:18
*** zhubingbing has quit IRC14:19
egonzalezAnyone have ever seen this error Field 'domain_id' doesn't have a default value in keystone logs?14:19
openstackgerritzhubingbing proposed openstack/kolla-ansible: Add freezer ansible role  https://review.openstack.org/40222414:20
openstackgerritcaoyuan proposed openstack/kolla-ansible: Fix the undefined error for vmtp_vm_availability_zone  https://review.openstack.org/42571514:21
*** absubram has joined #openstack-kolla14:22
*** zhubingbing has joined #openstack-kolla14:22
*** absubram_ has joined #openstack-kolla14:26
*** zhubingbing has quit IRC14:26
*** absubram has quit IRC14:27
*** absubram_ is now known as absubram14:27
openstackgerritMerged openstack/kolla-kubernetes: Gate fix for horizon and cinder  https://review.openstack.org/42436814:29
openstackgerritMerged openstack/kolla-kubernetes: Update docs to pin K8s at v1.5.1 and remove helm pull workaround  https://review.openstack.org/42505914:30
openstackgerritMerged openstack/kolla-kubernetes: Fix ci gate deploy multiple k8s slave  https://review.openstack.org/42408614:30
mliimaguys, need reviews here https://review.openstack.org/#/c/424158/614:31
*** shardy has quit IRC14:35
*** eaguilar has quit IRC14:40
*** kbaegis has joined #openstack-kolla14:40
*** Jeffrey4l has quit IRC14:44
*** Jeffrey4l has joined #openstack-kolla14:44
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfigure action for telegraf  https://review.openstack.org/42420214:47
*** jtriley has joined #openstack-kolla14:50
*** hfu has quit IRC14:51
*** tonanhngo has joined #openstack-kolla14:52
*** tonanhngo has quit IRC14:53
*** tonanhngo has joined #openstack-kolla14:53
*** rushil has joined #openstack-kolla14:54
*** zioproto has quit IRC14:55
*** jrist has quit IRC14:56
*** rwellum has quit IRC14:57
*** rwellum has joined #openstack-kolla14:57
*** tonanhngo has quit IRC14:58
des_consoladohey `git show d3d4a933` shows that heka has been deprecated and fluentd is being used. I've done `pip install -U kolla-ansible kolla` and then `kolla-ansible upgrade -i myinventory`, however the heka containers are still there on the network and compute nodes and I don't see any fluentd containers anywhere D:14:59
*** simon-AS5591 has quit IRC14:59
egonzalezJeffrey4l: dind't know whe were testing upgrades in gates ;) http://logs.openstack.org/46/425446/5/check/gate-kolla-ansible-dsvm-deploy-centos-binary-centos-7-nv/6ca1541/console.html#_2017-01-26_13_52_09_65199015:00
kbaegisdes_consolado: Unless you're willing to d15:00
kbaegisdes_consolado: You could do a kolla-build15:00
*** rushil has quit IRC15:00
egonzalezdes_consolado: fluentd was merged this week, pip tag is not made yet15:01
Jeffrey4legonzalez, actually, it just run upgrade action. it is not a real upgrade test15:02
des_consoladoah right, in git tag in the repository I see 4.0.0.0b1 and 4.0.0.0b2, I guess the kolla-ansible in pip is the b2 and the master tree isn't in pip yet15:03
egonzalezdes_consolado: yep15:04
*** lrensing has joined #openstack-kolla15:04
egonzalezdes_consolado: if want to use master better clone git repo and execute task from tools folder15:04
*** rushil has joined #openstack-kolla15:04
egonzalezpomac: ping15:05
*** absubram has quit IRC15:06
pomacegonzalez: sorry, been cleaning racks, racking hardware and... meetings.. =)15:09
egonzalezwere you able to fix the domain_id field in keystone db?15:09
*** jrist has joined #openstack-kolla15:09
pomacegonzalez: well, yes and no, it started working again but eventually failed - i suspect that it's a bug15:10
egonzalezhaving the same issue while doing upgrade from N to master15:10
pomacegonzalez: we have it in N15:10
egonzalezif only using a db_sync all works fine, if doing with zero downtime method fails15:11
*** lamt has joined #openstack-kolla15:11
pomacegonzalez: domain_id is cleared in some places for, to me, unknown reasons15:11
egonzalezasked at keystone channel but nobody replies15:11
pomacegonzalez: i suspect that we have the failure all the time now, the change we've done is to enable keystone v2 users (ie, set keystone v2 default domain to something != 'default'15:12
pomacegonzalez: have you played with changing the mtu btw? that's also "fun" ;)15:15
egonzalezpomac: i'll send a mail to the ML and see if someone explains or give a hit on how to proceed15:15
pomacegonzalez: good, lemme know how it goes =)15:15
*** hfu has joined #openstack-kolla15:16
kbaegisSo can someone explain how this works? (MariaDB container)—<ovs>—<kglobals nic>15:17
kbaegisIs that accurate? We're pinning the container directly to the nic, not through an intermediary bridge?15:17
kbaegisOr is it (MariaDB)—<docker0>—<ovs>—<nic>?15:17
kbaegisNeed to know, because I can't get: haproxy, mariadb, etc pinned to the vip15:18
egonzalezkbaegis: net host is used for containers15:18
egonzalezphysical nic IP == container IP15:18
kbaegistyvm15:19
kbaegisGonna try disabling haproxy and set the IP to the primary nic.15:21
*** lamt has quit IRC15:21
kbaegisAIO15:21
egonzalezkbaegis: haproxy vip need to be other than the interface IP15:22
egonzalezkbaegis: my globals.yml as example http://paste.openstack.org/show/596606/15:23
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla: Remove all kolla-ansible related docs  https://review.openstack.org/42574915:23
*** sacharya has joined #openstack-kolla15:31
*** kbaegis has quit IRC15:33
*** fguillot has joined #openstack-kolla15:35
*** stevemar has quit IRC15:36
*** kbaegis has joined #openstack-kolla15:37
*** stevemar has joined #openstack-kolla15:37
*** rushil has quit IRC15:39
*** crushil has joined #openstack-kolla15:40
*** matrohon__ has quit IRC15:41
openstackgerritcaoyuan proposed openstack/kolla-ansible: Optimize reconfigure action for telegraf  https://review.openstack.org/42420215:43
*** shardy has joined #openstack-kolla15:46
*** openstackgerrit has quit IRC15:48
*** sayantani01 has joined #openstack-kolla15:59
*** egonzalez has quit IRC16:00
*** openstackgerrit has joined #openstack-kolla16:01
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla-ansible: Remove image building related docs from kolla-ansible  https://review.openstack.org/42576616:01
*** egonzalez has joined #openstack-kolla16:01
sbezverkSamYaple: ping16:02
*** srwilkers has quit IRC16:09
*** krtaylor has quit IRC16:09
*** jrist has quit IRC16:10
kbaegisSo does the docker net=host override whatever docker is configured to do by default?16:17
*** papacz has quit IRC16:17
kbaegisi.e.- if docker is using the bridge networking, does kolla override that behavior?16:17
egonzalezkbaegis: http://www.dasblinkenlichten.com/docker-networking-101-host-mode/16:17
*** mdnadeem has quit IRC16:19
*** jrist has joined #openstack-kolla16:21
*** krtaylor has joined #openstack-kolla16:22
kbaegisegonzalez: So under /etc/conf.d/docker I just remove the networking config, correct? It didn't like —net=host16:24
kbaegisegonzalez: Or should kolla declare this automatically at run time?16:25
*** msimonin has quit IRC16:28
egonzalezkbaegis: im not getting what you mean. Don't want to use net-host or is not working?16:28
kbaegis*sigh* MariaDB isn't passing connectivity checks. I'll post a docker inspect in a second16:28
kbaegisHere's an inspect for my MariaDB16:29
kbaegishttps://hastebin.com/odezinitoh.json16:29
kbaegisThe network id correlates to my docker network "host"16:30
kbaegisSomethings not deploying right with the ansible scripts. I'm trying to figure out how to fix it16:31
egonzalezprechecks ensure no other service is using the port with the same IP mariadb is going to use16:31
egonzalezcan you share globals and ip a in the hosts?16:32
kbaegishappily16:32
*** prameswar has joined #openstack-kolla16:34
kbaegisGlobals: https://hastebin.com/ebiquxonut.bash16:34
kbaegisip addr: https://hastebin.com/igiqiragov.pl16:35
kbaegisre-running prechecks16:35
*** hfu has quit IRC16:36
*** hfu has joined #openstack-kolla16:37
*** hfu has quit IRC16:37
*** hfu has joined #openstack-kolla16:38
*** hfu has quit IRC16:38
*** hfu has joined #openstack-kolla16:39
*** hfu has quit IRC16:39
*** hfu has joined #openstack-kolla16:39
*** hfu has quit IRC16:40
kbaegispasses prechecks just fine. But still can't connect to mariadb as indicated. The port isn't occupied.16:40
kbaegisSame issue, even if I enable the vip and change to 192.168.254.416:41
SamYaplesbezverk: pong16:41
*** sayantani01 has quit IRC16:41
*** sayantani01 has joined #openstack-kolla16:42
*** srwilkers has joined #openstack-kolla16:42
sbezverkSamYaple: thanks for reply.. I wanted to check with you about one problem I hit in kolla-kube and interested how kolla dealt with it..16:43
*** zioproto has joined #openstack-kolla16:43
SamYapleok16:43
sbezverkSamYaple: it turned out that manage_db fails if it was already run against the database16:44
kbaegisegonzalez: Any clue? Passed all 167 prechecks16:44
SamYaplesbezverk: what service?16:44
sbezverkSamYaple: when I remove a service package example cinder it removes only bits, not database or service endpoints from keystone16:44
sbezverkSamYaple: so when I try to deploy cinder package again which includes bootstrap steps it fails as db manage goes banans16:45
kbaegisJust also verified that inside the container I can ping out16:45
SamYaplesbezverk: well, for one. kolla-* has never supported removing services in any fashioin16:45
SamYaplesbezverk: or removing/scaling down anything for the most part. though you _can_ scale down some things, its more of a conincendence rather than design16:46
zioprotoI have a question, I added two hosts in the [network] group. I have now two HA proxy instances with VRRP and that is fine. But what I really want is to have two L3-agent that work at the same time, as I have now in production. Is this is a possible scenario when deploying with Kolla ?16:46
sbezverkSamYaple: ok got..16:46
sbezverk SamYaple: I plan to introduce then a set of chart for cleaning up leftovers16:46
SamYaplesbezverk: with kubernetes it will be esiear to do this for sure. since its a contastnaly running thing that can track up/down/delete actions. where ansible is point-in-time16:47
egonzalezzioproto: enable_neutron_agent_ha16:47
sbezverkSamYaple: since kolla has no already proposed solution16:47
zioprotoegonzalez, but I dont want ha. I just want multiple L3-agents like I have now :)16:47
zioprotowe have a big production cloud16:47
zioprotoit is not possible to ahve all virtual name spaces16:47
zioprotoin a single bare metal server16:48
zioprotoso we scale it using multiple l3-agent16:48
zioprotoworking at the same time in parallel16:48
zioprotonot in HA16:48
zioprotowhen you create a router neutron-server selects on which network node schedule the neutron router in a round robin fashion16:48
zioprotothis is how neutron works16:48
des_consoladohey is there some docs/guidance anywhere on how to add to kolla-ansible if there aren't currently plays/configuration for something you want to use? Or I guess is it just a work it out for yourself and submit a PR kinda thing?16:48
kbaegiszioproto: Are you doing DVR?16:48
zioprotokbaegis, no, no DVR16:48
des_consolado(I need to use the BGP dynamic routing plugin - Ryu - doesn't look like this is supported by kolla-ansible right now)16:49
zioprotoI am doing standard network node with VXLAN16:49
zioprotobut I have two network nodes16:49
zioprotoso when I do 'neutron agent-list' I see two l3 agents16:49
des_consoladoIt doesn't look that difficult I guess, you just add some vars and then some j2 stuff in the templates dir and then tasks in the tasks16:49
kbaegiszioproto: Oh. So they're a congestion point for all e-w, and n-s traffic?16:49
zioprotoand my qrouter- namespaces16:49
zioprotoyes they are a congestion point16:49
zioprotobut because I can add many of them16:49
zioprotoI dont care16:49
zioprotothey are not in HA16:50
zioprotothey are just many16:50
zioprotonow16:50
zioprotowe have everything with puppet16:50
zioprotowe want to migrate to kolla16:50
zioprotokill our puppet layer that we also use to deploy16:50
zioprotobut16:50
zioprotoI have to understand how to make the same setup16:50
zioprotoI have already with stable/newton kolla deployed the example multinode scenario16:50
zioprotoI am trying to understan16:50
zioprotowhat changes I need to the inventory16:50
zioprototo match my existing cloud16:51
zioprotoI dont get how to have something with more than 1 network node running the l3-agent, not in HA16:51
zioprotoHA is just a feature in neutron.conf, you can have also multiple l30agent without HA16:51
*** adrian_otto has joined #openstack-kolla16:52
zioprotoI will try enable_neutron_agent_ha but I expect then that neutron routers are created with HA yes16:52
zioprotoand this breaks IPv6 in my cloud16:52
*** saneax is now known as saneax-_-|AFK16:52
kbaegiszioproto: So this is all documented in ansible/inventory/16:53
*** lamt has joined #openstack-kolla16:53
kbaegisthe [network] inventory states "The network nodes are where your l3-agent and loadbalancers will run16:53
zioprotoyes, and I added two hosts in [network]16:54
kbaegisIf you just want l3-agent, you can alter the [network]16:54
kbaegiszioproto: I think that will give you parity16:54
zioprotobut only on 1 I see the l3-agent running16:54
zioprotoI mean the other is a backup16:54
zioprotoI am deploying again with enable_neutron_agent_ha = yes16:55
kbaegisI think ha is 1:1 by default16:55
kbaegisas in failover16:55
*** athomas has quit IRC16:56
kbaegisegonzalez: Any idea why my containers aren't attaching? Do I need to create some custom iptable rules or something?16:57
egonzalezzioproto: as far I can see there is no official way to add multiple agents in the same node, as they share rpc name16:58
kbaegiszioproto: Were you configuring your hypervisors to round-robin to the two separate network nodes?16:58
zioproto]yes16:58
zioprotoI have two diffent network nodes16:58
kbaegisegonzalez: Not two separate agents. He was leveraging the hypervisors multiple route option with the rr packet scheduler16:59
kbaegisI don't think kolla knows how to do that currently16:59
zioprotokbaegis, not sure about what you are saying16:59
zioprotoso I have17:00
zioprotoin my current prod infrastructure17:00
zioprototwo servers17:00
zioprotowith tho different host names17:00
zioprotoboth running l3-agent17:00
*** saneax-_-|AFK is now known as saneax17:00
zioprotoI have two l3-agent with two different UUID17:00
zioprotoI can even migrate the qrouter- namespaces from 1 server to the other server17:00
kbaegiszioproto: That's not the issue. The issue is how your vm hosts are sending packets to those two servers17:00
egonzalezzioproto: yup, but have created two different l3-agent files and started each service with his own config file17:00
*** jrich523 has quit IRC17:01
zioprotoegonzalez, yes of course, because they are on different servers17:01
kbaegisThe default is to use one desination and gobble up the l3 address as it fails17:01
*** MasterOfBugs has quit IRC17:01
kbaegisFor load balancing, you need a packet scheduler to two distinct addresses17:01
*** pramodrj07 has quit IRC17:01
*** PramodJ has joined #openstack-kolla17:01
*** MasterOfBugs has joined #openstack-kolla17:01
kbaegisOtherwise you have HA, but no load balancing17:01
zioprotookay17:02
zioprotowhere is the correct place for operators to bring feedback about Kolla ? at the PTG ? or at the Milano Ops MIdcycle ?17:02
*** jrich523 has joined #openstack-kolla17:02
egonzalezzioproto: summit i guess, i dont think any from kolla to be at Ops midcycle at milano17:04
*** inc0 has joined #openstack-kolla17:04
zioprotosummit you mean Boston, right ?17:04
egonzalezand PTG is more designed to be for developers, summit is for operators17:05
egonzalezyeah, boston17:05
*** kbaegis has quit IRC17:05
inc0o/17:05
*** tonanhngo has joined #openstack-kolla17:05
inc0egonzalez, I couldn't work yesterday, my mariadb is broken for reasons unknown17:05
*** kbaegis has joined #openstack-kolla17:06
inc0sorry:(17:06
egonzalezi'm working on it but stuck with errors in keystone database after upgrade, everything works fine but create user don't17:09
egonzalezinc0: Field 'domain_id' doesn't have a default value17:09
inc0yay17:09
*** tonanhngo has quit IRC17:10
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible: Refactor register.yml files  https://review.openstack.org/41604617:10
*** crushil has quit IRC17:11
inc0Jeffrey4l, did you see anything strange lately in mariadb?17:14
Jeffrey4linc0, no. what did u find?17:14
inc0I tried to deploy yesterday and it doesn't work well17:14
*** sayantani01 has quit IRC17:14
inc0on my ubuntu at least17:14
inc0fresh master, fresh vms17:14
Jeffrey4linc0, which task failed?17:14
inc0it doesn't bind to port17:14
inc0so building cluster failed17:14
Jeffrey4lhrm. never saw this.17:15
inc0first error is "Connection refused"17:15
inc0so problem is before that17:15
inc0and I have no idea what causes it:/17:15
Jeffrey4linc0, bootstrap container failed?17:15
inc0no17:15
inc0starting contianers work17:15
inc0but waiting for ready times out17:16
Jeffrey4lkeepalived/vrrp issue?17:16
sbezverkinc0: qq is there a way to drop db using "ansible localhost -m mysql_db"17:17
inc0negative, local ss -plant doesn't show it17:17
inc0sbezverk, yeah, should be17:17
inc0no idea tbh17:17
sbezverkinc0: who might know? ideas?17:17
inc0worst case scenerio you can do ansible -m shell -a "mysql -c 'DROP DATABASE'"17:18
Jeffrey4lsbezverk, add: state=absent?17:18
inc0or whatever syntax is17:18
sbezverkinc0: that was my plan Z ;)17:18
Jeffrey4lsbezverk, check "ansible-doc mysql_db"17:18
kbaegis Anyone know how I can have a perfectly functional MariaDB container that can ping out, but not have it pass the network connectivity check?17:18
sbezverkJeffrey4l: thank you will do, I did not know it was a standard interface17:18
kbaegisI need some help with this17:18
kbaegisP.S. - the port passes the pre-check17:19
inc0kbaegis, I saw something similar yesterday17:19
SamYaplesbezverk: its not a standard interface, but it is a standard convention that most people follow17:19
inc0only not perfectly functioning17:19
inc0since port hasn't been listened on17:19
kbaegisinc0: Inside the container I can connect the client to the server. Outside, it wasn't working17:19
kbaegisThe container can ping a public IP too17:20
inc0that is...strange since it has host network17:20
jascott1morning folks17:20
inc0 /etc/hosts difference? It shouldn't be the case as it really uses usual one from host17:20
inc0but copies it so strange17:20
kbaegisI do have one address hard coded into that17:21
kbaegisLet me check17:21
sbezverkJeffrey4l: looks like drop db is not implemented, I do not see it in the doc, but I do see create.. strange17:22
kbaegisIt did copy17:22
*** sayantani01 has joined #openstack-kolla17:24
Jeffrey4lsbezverk, it is implemented https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/database/mysql/mysql_db.py#L32717:24
*** unicell has joined #openstack-kolla17:26
inc0http://paste.openstack.org/show/596623/ <- mariadb non-master logs17:26
sbezverkJeffrey4l: I see, strange they do not mention in the doc, thanks for poiting out..17:26
*** lamt has quit IRC17:27
kbaegisinc0: So I'm able to nc to the localhost:330617:27
Jeffrey4lnp :)17:27
kbaegiswhere mariadb is running, and I get a connection string17:27
*** eaguilar has joined #openstack-kolla17:27
kbaegisThis pretty clearly falls on the connection checking script. Unless it's not supposed to be using port 3306?17:28
inc0yeah I have something similar17:28
inc0on multinode17:28
*** lamt has joined #openstack-kolla17:28
inc0only it's more than failing on conenctions tring as cluster members fails to connect to17:28
inc0o17:28
inc0too17:28
zioprotoegonzalez, I deployed with enable_neutron_agent_ha and looks much better. Also I created a router and the router has ha = false as I wanted. Now I am trying to look for the qrouter-<uuid> namespace, where is that supposed to be created ? On the baremetal node hosting the docker container I cant find it. Is it inside a docker container ?17:29
kbaegisinc0: Weird. Well if the port is exposed and the versions are compatible it's got to be networking?17:29
inc0no idea, telnet from other node worked perfectlyu17:30
inc0which distro you're using?17:30
kbaegiszioproto: Oh yeah. If the l3-agents are sharing the same queue, that needs to be replicated too17:30
*** tonanhngo has joined #openstack-kolla17:31
inc0http://paste.openstack.org/raw/596624/ this is my master node logs17:31
inc0I'm baffled17:31
*** eaguilar has quit IRC17:32
inc0nc works from everywhere, mariadb itself doesn't17:35
inc0funny enough I have lots of connections from everywhere, so it's somethign about cluster building17:36
*** rwellum is now known as Penfold17:36
inc0hmmm.....galera is supposed to create it's database right?17:38
inc0for wsrep?17:38
SamYapleinc0: its straight forward17:38
SamYaplecheck /var/lib/mysql/xtrabackup.log17:38
SamYapleyoull see the error17:38
SamYaplemight not be called xtrabackup, maybe innodb somethign17:39
SamYapleanyway, its a log file in /var/lib/mysql17:39
inc0Error executing 'FLUSH ENGINE LOGS': DBD::mysql::db do failed: Can't execute the query because you have a conflicting read lock at /usr//bin/innobackupex line 3059.17:40
inc0170126 17:20:59  innobackupex: Waiting for ibbackup (pid=1357) to finish17:40
SamYaplewell there you have it17:40
SamYapleyouve already got another process running17:40
SamYaplecheck the futher up logs to see if that crashed17:40
kbaegisHuh.17:40
*** Penfold_ has joined #openstack-kolla17:40
*** hfu has joined #openstack-kolla17:41
*** prameswar has quit IRC17:41
*** eaguilar has joined #openstack-kolla17:41
inc0ok, what the actual fuck...I indeed see mariadb process in host pid namespace17:42
SamYapleuhhh inc0. thats how it works17:42
SamYapleyou can see all processes from the host pid namespaces17:42
inc0ok17:43
SamYaplebut from the mariadb pid namespaces you cant see the host processes17:43
inc0yeah, that's fine then17:43
*** hfu has quit IRC17:43
inc0but yeah, I have both mysql_safe and mysqld running in container17:44
inc0something is seriously wrong17:44
SamYaplethats also how that works17:45
SamYaplemysqld_safe is a shell script that launchs mysqld17:45
SamYapleso far so normal17:45
*** eaguilar has quit IRC17:45
SamYaplethat log makes it pretty clear to me whats going on. where is the confusion?17:46
kbaegisSamYaple: What's responsible for setting the mutex?17:46
kbaegisSamYaple: Is that karbor?17:47
inc0SamYaple, confusion is, what is the other process17:47
inc0this is pristine fresh ubuntu 1617:47
SamYapleinc0: innodbbackup.17:47
SamYapleinc0: whats running at pid=1357?17:47
SamYaplekbaegis: the lock is set by xtrabackup in prep for recieving or sending an IST or SST17:48
*** Penfold_ has quit IRC17:48
inc0nothing now oO17:48
SamYaplekbaegis: it looks like multiple xtrabackups are running for some reason17:48
inc0yeah17:48
inc0I'll redeploy and see what'17:48
inc0what's happening17:48
*** zioproto has quit IRC17:49
inc0maybe it's just this innodb backup being so slow that stuff times out17:49
SamYaplethe timeout times were set to 900 in all of the startup scripts for that reason17:49
SamYaplethe official startup scripts taht is17:49
openstackgerritSteven Dake proposed openstack/kolla-kubernetes: Test keystone endpoint list  https://review.openstack.org/42580917:50
kbaegisI haven't looked too far into Karbor, but isn't that precisely what it does? Backs up the mariadb tables?17:51
kbaegisBut that gets deployed later?17:51
*** sambetts is now known as sambetts|afk17:51
*** egonzalez has quit IRC17:52
*** shardy is now known as shardy_afk17:53
*** david-lyle has quit IRC17:59
kfox1111morning.17:59
inc0SamYaple, btw did you get my msg about kolla-salt being good to go18:00
inc0?18:00
*** mgoddard has quit IRC18:02
*** lamt has quit IRC18:02
inc0SamYaple, no such process:/ seems like it finished but lock hasn't been lifted18:02
sdakeanyone ever seen office space?18:04
* sdake groans18:04
rhalliseysdake, where's my stapler?18:04
sdakerhallisey REDLINE18:04
*** unicell has quit IRC18:04
kfox1111hmm...18:04
kfox1111was close to working this time: http://logs.openstack.org/periodic/periodic-kolla-kubernetes-deploy-centos-binary-2-ceph/c22108a/console.html18:05
*** rmart04 has quit IRC18:05
inc0ok, gonna grab some food and to debug I go18:07
kfox1111sdake: does catalog list work?18:07
sdakekfox1111 i can try18:07
sdakemoment18:07
inc0kbaegis, let me know if you find anything on your end18:07
kfox1111(just curious if endpoint list is an admin feature)18:07
kfox1111catalog list isnt.18:07
kfox1111if catalog list works but endpoint list doesnt, I know what the problem is.18:07
sdakekfox1111 yes catalog list works18:07
sdakeglorious18:07
*** Serlex has quit IRC18:08
kfox1111and... another merge conflict. :/18:08
kfox1111sdake: the fix is in here: https://review.openstack.org/#/c/418550/18:08
kfox1111Basically, we never do any keystone admin only commands in the gate,18:09
sdakekfox1111 the create-endpoints yaml?18:09
kfox1111and keystone-admin endpoint was setup to be internal only.18:09
sdakekfox1111 i just added one18:09
kfox1111the create-endpoints patch and the iscsi_workflow one.18:09
kfox1111sdake: it will fail though.18:10
sdakehttps://review.openstack.org/#/c/425809/18:10
sdakekfox1111 roger18:10
kfox1111k.18:10
sdakekfox1111 can you breka out just those two changes18:10
SamYapleinc0: yea i saw the kolla-salt message18:10
SamYapleinc0: im not sure about the "lock hasn't been lifted" business18:10
sdakekfox1111 into a parent of that other patch - so we can get that fix merged18:10
sdakekfox1111 super patches like the one you have are hard to follow18:10
kfox1111yeah. probably time to break it up.18:11
sdakekfox1111 if you do that, I can rebae the dev env on top of it18:11
kfox1111kept getting 99% of the way there, then finding one more thing.18:11
sdakeyup18:11
*** unicell has joined #openstack-kolla18:11
kfox1111it works for trunk now all except cinder's broken.18:11
kfox1111haven't figured out why though.18:11
kfox1111suspecting bad containers. but not sure yet.18:11
*** eaguilar has joined #openstack-kolla18:12
kfox1111the cinder volumes don't attach to nova properly.18:12
kfox1111everything else seems happy though.18:12
*** Syffs has joined #openstack-kolla18:13
*** crushil has joined #openstack-kolla18:15
*** eaguilar has quit IRC18:16
sbezverkkfox1111: ping18:17
kfox1111sbezverk: ping18:17
sbezverkkfox1111: please check this bp https://blueprints.launchpad.net/kolla-kubernetes/+spec/helm-service-cleanup18:17
*** gfidente is now known as gfidente|afk18:18
kfox1111sbezverk: interesting problem.18:18
kfox1111not sure the correct behavior though.18:19
kfox1111it is 'handy' in one way for it to just update the records on the new run.18:19
kfox1111it also is a foot gun someone can unexpectedly step on.18:19
sbezverkkfox1111: is it not everything?18:20
sbezverkkfox1111: but man I am tired of manually cleaning up mariadb and keystone18:20
kfox1111Perhaps, we should make some cleanup scripts/jobs so that if someone wants to change things, they can, but normally it will prevent random accidental reconfigures?18:20
kfox1111I do want the ability to launch a service package without jobs too though.18:21
kfox1111so maybe thats the other way to solve it?18:21
sbezverkkfox1111: I will push PS with cinder cleanup service18:21
sbezverkas a prototype18:21
kfox1111someone might accidently delete cinder, say,18:21
kfox1111and wants to load it back, without doing any db/keystone changes.18:21
sbezverkit will include 3 jobs drop db drop service in keystone and user in keystone18:21
kfox1111that would allow that.18:21
sbezverkit will not be a part of main cinder package18:22
kfox1111yeah. sounds good.18:22
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: Local registry in gate for trunk builds  https://review.openstack.org/41855018:22
inc0SamYaple, I think it has been lifte18:23
inc0d18:23
inc0just something failed to move forward18:23
sbezverkI have two options either modify existing _common_create_db to accept a parameter operation where delete would be pass to it18:23
inc0lslocks shows nothing18:23
sbezverkor use _common_delete_db macro18:23
sbezverkkfox1111: what would be your choice?18:23
kfox1111hmm... let me look.18:25
kfox1111ah. that ones still ansible based. :/18:25
*** unicell has quit IRC18:25
sbezverkkfox1111: yeah at least for db management18:26
kfox1111on the fence. sharing code woudl be nice.18:26
kfox1111but, knowing that the code your about to run couldn't possibly delete your database is also nice.18:27
sbezverkkfox1111: I tend to use a separate macro because of this specific reason18:27
kfox1111yeah.18:27
sbezverkespecially during the debug phase ;)18:27
kfox1111+1 :)18:27
kfox1111ok. lets keep it seperate for that reason.18:27
sbezverkkfox1111: sounds good..18:28
kfox1111sbezverk: you were right. bzip wasn't in the vm.18:29
*** unicell has joined #openstack-kolla18:30
sbezverkkfox111: yeah it is just I hit it bunch of times in the past but thought myabe it has made into the default package list18:31
kfox1111yeah.18:31
kfox1111the periodic jobs did run though. :)18:31
kfox1111http://logs.openstack.org/periodic/periodic-kolla-kubernetes-deploy-centos-binary-2-ceph/c22108a/console.html18:31
kfox1111just gotta fix it up slightly, and tomorrow maybe we'll have some tarballs. :)18:32
sbezverkkfox1111: cool18:32
*** msimonin has joined #openstack-kolla18:33
inc0https://review.openstack.org/#/c/424784/ can I have core eyes on this please?18:34
inc0so we can get this process out of the way?18:34
*** v1k0d3n has quit IRC18:35
*** jtriley has quit IRC18:36
*** lamt has joined #openstack-kolla18:37
*** lamt has quit IRC18:37
*** msimonin has quit IRC18:39
*** msimonin has joined #openstack-kolla18:39
*** msimonin has quit IRC18:40
sdakekfox1111 can you tell me when you will have that gate fix patch split up so I can continue on the dev env?18:40
kfox1111sdake: in process18:40
*** msimonin has joined #openstack-kolla18:40
sdakekfox1111 cool thanks :)18:40
kfox11115 min18:40
*** unicell has quit IRC18:40
*** msimonin has quit IRC18:41
*** msimonin has joined #openstack-kolla18:41
*** msimonin has quit IRC18:41
*** msimonin has joined #openstack-kolla18:42
*** msimonin has quit IRC18:42
*** eaguilar has joined #openstack-kolla18:42
*** msimonin has joined #openstack-kolla18:42
*** lamt has joined #openstack-kolla18:43
*** msimonin has quit IRC18:43
*** v1k0d3n has joined #openstack-kolla18:43
*** msimonin has joined #openstack-kolla18:43
*** msimonin has quit IRC18:44
inc0who wants to be fuel ptl? spot is still open:)18:47
*** eaguilar has quit IRC18:47
srwilkerslol18:47
openstackgerritMerged openstack/kolla-ansible: Refactor register.yml files  https://review.openstack.org/41604618:48
*** crushil has quit IRC18:49
kfox1111might be the best way to get fuel using kolla-kubernetes. ;)18:50
kfox1111and to get a slick installer for kolla-kubernetes. :)18:50
SamYapleyou have to have a commit in the fuel repos to be PTL18:50
inc0yeah I know18:50
inc0we should commit doc patch to all the installers18:51
openstackgerritEduardo Gonzalez proposed openstack/kolla: Implement Keystone zero-downtime upgrade  https://review.openstack.org/42583318:51
inc0and ninja-self-nominate18:51
SamYaplei wrote a script that finds all the projects that ahvent had people with PTL subbmissions. i was going to self nominate after merging a doc patch18:51
inc0and make them use Kolla18:51
SamYaplethat was the plan in the paser :P18:51
SamYaplepast*18:51
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: Keystone-admin external support  https://review.openstack.org/42583618:52
kfox1111sdake: that should it.18:52
*** unicell has joined #openstack-kolla18:52
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: Local registry in gate for trunk builds  https://review.openstack.org/41855018:54
kfox1111inc0: the trunk patch isn't huge anymore. :)18:54
kfox1111inc0: the only thing stopping it from merge is someone with some time figuring out why volumes wont attach. it is really close.18:54
sdakethanks kfox111118:55
inc0kfox1111, can't do now, not until I figure out this mariadb scary thign18:55
sdakethanks sbezverk :)18:55
kfox1111inc0: k.18:56
kfox1111inc0: I'll keep poking at it off and on as I get time.18:56
*** Penfold has quit IRC18:57
*** Penfold has joined #openstack-kolla18:58
*** klindgren__ is now known as klindgren19:00
*** harlowja has quit IRC19:02
kfox1111inc0: is it bzip package name on ubuntu?19:05
*** crushil has joined #openstack-kolla19:05
inc0kfox1111, bzip119:06
inc0bzip2**19:06
kfox1111k. thanks. :)19:06
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: Fix for the periodic kolla-kubernetes processes.  https://review.openstack.org/42584719:09
sdakekfox1111 so quick quesiton:19:13
sdakekfox1111 you said this patch should include just keyseond end point and ceph_iscsi:https://review.openstack.org/#/c/425836/19:13
sdakekfox1111 however it has a bunch of stuff19:14
kbaegisinc0: Will do. Just got back from brunch. I'll keep looking19:14
kfox1111artifactg of splitting the way I did. :/19:15
kfox1111I should amend the commit message.19:15
kfox1111there's a fix for a network start race condition and some cleanup work for the workflow scripts, that the keystone_endpoint thing forced.19:16
kfox1111I really don't want to split it into 4 ps's though.19:16
*** absubram has joined #openstack-kolla19:18
kbaegisinc0: Did you try setting the timeout to something ridiculous? Like 30 minutes?19:19
kbaegisOr is it just a stale write mutex across the whole mariadb instance?19:19
kbaegisUgly19:20
*** unicell has quit IRC19:20
*** unicell has joined #openstack-kolla19:20
kfox1111sdake: actually, rereading through it, the other bits are mostly about ensuring the job startups will work, as the admin port is now external, so network has to be started before the jobs now.19:23
kfox1111I'll amend the commit log to have that though.19:23
sdakekfox1111 - ok - sounds good19:24
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: Keystone-admin external support  https://review.openstack.org/42583619:26
kfox1111dang...19:27
kfox1111I really hoped it was the docker_registry name thing that would fix the cannot overwrite digest bug... :/19:28
kfox1111oh well.19:28
kfox1111the cached containers should fix it.19:29
*** saneax is now known as saneax-_-|AFK19:31
kfox1111inc0: yeah, I think there may be something wrong with the libvirt trunk container.19:32
kfox1111not so much volume attach goodness.19:33
*** Alexey_Y has joined #openstack-kolla19:33
SamYaplejust fyi, the libvirt container changed a good bit with the static uid/gid patch and i can assure you wasn't fully tested before merging19:34
SamYapleso id blame that always19:34
*** jheroux has quit IRC19:35
kfox1111have a link for that?19:36
*** adrian_otto has quit IRC19:36
kbaegisinc0: So I found this: ./group_vars/all.yml:mariadb_wsrep_port: "4567" versus ./group_vars/all.yml:database_port: "3306"19:38
kbaegisIs that expected?19:38
kfox1111I think the ports should be different.19:38
kbaegisGotcha. I don't understand the xtrabackups functionality19:39
*** crushil has quit IRC19:39
kbaegisI'll need to dig further into that. The ports up though.19:40
kfox1111if I remember correctly, there is a port for the db as normal, a port for cluster replication traffic, and a port for rsync raw data during initial recovery.19:40
kfox1111thought I haven't looked at how kolla implemented it.19:40
kfox1111oh. and a port ofr cluster health if you implement one.19:41
*** nihilifer has quit IRC19:42
*** nihilifer has joined #openstack-kolla19:43
kbaegisWell we have an issue where ansible deploy isn't able to talk to mariadb. The port's open, netcat verifies19:43
kbaegisI just tried removing the xtrabackup in wsrep-notify19:43
*** david-lyle has joined #openstack-kolla19:45
kfox1111very interesting... I was contimplating trying to deploy k8s in k8s. http://blog.kubernetes.io/2017/01/how-we-run-kubernetes-in-kubernetes-kubeception.html19:45
kfox1111though with helm. :)19:45
openstackgerritSteven Dake proposed openstack/kolla-kubernetes: WIP: Add Development Setup Script for use with Kubernetes Helm  https://review.openstack.org/40572019:46
kbaegisIf I had 15 nodes, k8s would be awesome19:46
*** mgoddard has joined #openstack-kolla19:47
kfox1111it scales from 1 node on up. :)19:47
kbaegisNo one on #kubernetes could give me a straight answer on how to get k8s working on gentoo. Use minikube isn't really an answer19:47
kbaegisUse a VM to host kolla which can talk to the hosts hypervisor19:47
kbaegisSeems absurd19:48
sdakekbaegis try slack kubernetes channel19:48
sdakekbaegis kubernetes community not super active on irc19:48
*** MasterOfBugs has quit IRC19:49
*** PramodJ has quit IRC19:49
*** sayantani01 has quit IRC19:51
kfox1111kbaegis: yeah. slack channel is more responsive.19:52
kfox1111all their startup scripts are written for systemd though, and gentu's a holdout I hear.19:52
kfox1111so you may have to write those from scratch.19:52
kfox1111gentoo I mean.19:52
*** jheroux has joined #openstack-kolla19:53
inc0kbaegis, yeah, these ports are supposed to be different19:53
inc0somethings different happening there19:53
kbaegiskfox1111: Thank god Gentoo isn't on systemd19:55
kfox1111kbaegis: sorry to hear gentoo's not on systemd. ;)19:55
kbaegiskfox1111: No need to impugn Pottering OT here though19:55
kfox1111as a dev, its soooo much better to write startup scripts.19:55
kfox1111as an op, I haven't missed the old init much, other then the initial pain of reordering systemctl restart foo / service foo restart.19:56
kbaegisI agree. Systemd is the best operating system you can run underneath Linux19:56
kbaegisDid you know it has a web frontend?19:56
kfox1111the fact I can jump between ubuntu and centos and remember how to enable a service at reboot is awesome too.19:56
kbaegis(not kidding)19:57
kfox1111kbaegis: what doesnt. ;)19:57
kfox1111linux has had all sorts of web admin frontends for a long long time. :)19:57
kfox1111never use them. but I know they exist.19:57
sbezverkkfox1111: forgot to mention, iscsi pv/pvc is working now :-) there is no dynamic support though19:57
kfox1111sbezverk: cool. :)19:58
kbaegiskfox1111: It's the linux paradigm, do all things, and make them interdependent19:58
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes: WIP adding cleanup service for cinder  https://review.openstack.org/42587520:00
kfox1111kbaegis: heh. linux is pragmatic.20:02
kfox1111if the arguments for why systemd sucks were s/systemd/linux kernel/20:02
kfox1111people would suddently start saying "its ok"20:02
kfox1111the linux kernel "is monolythic", "puts way too much in one process", "wants to take over the world"...20:03
kfox1111;)20:03
kbaegisNot going to debate it. Systemd is a mess. I'm glad it works though20:03
kfox1111I agree with the unix phylosophy of making a tool do one thing, and doing it well.20:03
kfox1111thought at doesn't mean the tools cant be bundled up together. :)20:03
kfox1111anyway... :)20:04
kfox1111all I know is, my life has been simplified since it came along. so a net positive. I havent looked at the code, but don't really need to.20:05
kfox1111these days, systemd is the thing that gets k8s up engouh to deal with the rest in k8s. :)20:05
*** harlowja has joined #openstack-kolla20:09
kfox1111http://logs.openstack.org/36/425836/2/check/gate-kolla-kubernetes-deploy-centos-binary-2-external-ovs-nv/05106d7/console.html20:15
*** mgoddard has quit IRC20:15
kfox1111that one's really bugging me. :/20:15
kfox1111why has setup-resolv-conf suddently become unstable?20:15
kfox1111all its doing is passing resolv.conf back to the host.20:15
kfox1111I wonder if I should just rewrite that to be a helm job now, and have it directly push into the configmap. :/20:17
sbezverkkfox1111: I heard that dns fix is getting into 1.6 kube20:20
inc0sooo...I used stable/newton images20:20
inc0and it's working20:20
sbezverkso soon we will not need a workaround20:20
kfox1111resolvconf?20:20
inc0so we messed up something in images20:20
SamYaplekbaegis: mariadb + galera uses 4 ports, one for mysql traffic like normal (3306), one for wsrep (4567), one for system state transfers SST (4444) and one for incremental state transfers IST (4568)20:20
SamYapleinc0: probably the static uid/gid patch. https://review.openstack.org/#/c/412231/ there were kolla-ansible and probably kolla-kubernetes changes taht were supposed to go with that that i dont think people implemented20:21
SamYaplei ran out of time to work on it but someone else started working on it after i abanonded it20:22
kfox1111SamYaple: what changes have to be made on the kolla-ansible/kolla-kubernetes side?20:22
SamYaplekfox1111: that i know of, libvirt permissions, but they need to be syncd between centos and ubuntu20:22
SamYapleand then tehre is /dev/kvm permissions20:23
openstackgerritEduardo Gonzalez proposed openstack/kolla-ansible: Integrate designate with neutron  https://review.openstack.org/42467920:23
kfox1111permissions, like we specify the uid to use to the conatiner via env,20:23
kfox1111or something else?20:23
SamYaplelike socket and device permissions20:23
SamYapleand probably more.20:23
kfox1111hm...20:23
SamYaplelike i said, i ran out of time20:23
SamYapleso i stopped working on it20:23
kfox1111should the patch be reverted then?20:23
SamYaplenext thing i new, someone had restored it and merged it20:24
SamYapleknew*20:24
kfox1111(I think the patch is very important. but if broken...)20:24
inc0I'll try reverting this patch and see if it helps20:24
sdakehttps://www.openstack.org/assets/reports/OpenStack-2016-Annual-Report-final-draft.pdf20:24
sdakeSamYaple i restored it20:25
sdakenot sure why it was merged20:25
sdakeSamYaple klingren was going to take over the work20:25
sdakei said right in the review it wasn't ready20:25
sdakeSamYaple so its not your error20:25
SamYaplejust letting people know. i dont think it was ready for merge, but again, i ran out of time to work on it20:26
SamYaple_testing_ is what was needed20:26
SamYaplei got it passign the gate20:26
sdakeSamYaple roger - thank sfor getting it started ;)20:26
sdakeSamYaple yup sometimes manual testing > gate testing20:27
sdakesound slike the root cause was sorted out - yay ;)20:27
kfox1111sdake: thanks for the link.20:27
sdakekfox1111 i haven' read it yet20:27
sdakekfox1111 lemme know if anything is intereting in there :)20:28
kfox1111kk20:28
kfox1111"upstream community has been20:28
kfox1111very supportive of users and receptive20:28
kfox1111to feedback"20:28
kfox1111heh....20:28
kfox1111"We updated our20:29
kfox1111mission statement to increase focus20:29
kfox1111on interoperability and the cloud end20:29
kfox1111users needs."20:29
kfox1111I sure hope so.20:29
*** sayantani01 has joined #openstack-kolla20:31
sbezverkkfox1111: right resolve.conf20:36
*** gfidente|afk has quit IRC20:36
kfox1111sbezverk: nice. :)20:39
kfox1111I really hope it makes it. :)20:39
*** lrensing has quit IRC20:39
kfox1111ugg20:40
kfox1111"Known as the 'Composable Shell', or CSHELL"20:40
kfox1111thanks ms.... more confusion. :/20:40
*** lrensing has joined #openstack-kolla20:41
kfox1111nice...20:48
kfox1111sbezverk: sdake: https://review.openstack.org/#/c/425836/20:48
inc0I have theory what might have happened to mariadb20:51
kfox1111yeah?20:51
inc0https://review.openstack.org/#/c/412231/33/docker/macros.j2 we set up /var/lib/mysql as homedir20:52
inc0and do some stuff with it later on20:52
kfox1111and the uid stuff is messing with that?20:54
inc0it might20:55
inc0I'll test it out20:55
*** gema has joined #openstack-kolla20:56
kfox1111bbiab20:59
*** esmiurium has quit IRC20:59
*** jrobinson has joined #openstack-kolla21:12
*** crushil has joined #openstack-kolla21:14
*** gema has quit IRC21:22
*** unicell has quit IRC21:24
*** gema has joined #openstack-kolla21:29
*** mgoddard has joined #openstack-kolla21:30
*** sacharya has quit IRC21:34
*** l4yerffeJ has joined #openstack-kolla21:34
*** adrian_otto has joined #openstack-kolla21:34
*** Jeffrey4l_ has joined #openstack-kolla21:34
*** l4yerffeJ__ has quit IRC21:35
*** Jeffrey4l has quit IRC21:35
*** rmart04 has joined #openstack-kolla21:43
*** hfu has joined #openstack-kolla21:44
*** hfu has quit IRC21:47
*** jtriley has joined #openstack-kolla21:47
*** aric49 has joined #openstack-kolla21:51
jrobinsonsayantani01, hi this is Joe from the docs team, just replied to your email :)21:53
*** mgoddard has quit IRC21:54
*** jtriley has quit IRC21:54
sayantani01Hey jrobinson, I will check your email shortly. I am running something and am disconnected to the vpn at the moment.21:55
*** rhallisey has quit IRC21:56
jrobinsonno problems.21:56
*** rhallisey has joined #openstack-kolla21:56
*** rhallisey has quit IRC21:57
v1k0d3ninc0: sounds like kolla world dominance. nice.21:58
inc0?21:58
v1k0d3nfuel takeover21:59
inc0running for Fuel PTL? ofcourse it's world dominance!21:59
inc0it's also a joke;)21:59
*** breitz has quit IRC22:01
*** fguillot has quit IRC22:01
inc0but, last release projects that didn't have PTL candidate were threatened of being kicked off big tent22:01
*** breitz has joined #openstack-kolla22:01
inc0it's just an oversight probably tho, Fuel community shrank and probably they don't read as much ML as they used to22:02
v1k0d3nperhaps it's a shift in focus.22:02
inc0they did lay off a lot of people while at it22:03
inc0they being Mirantis22:03
openstackgerritMark Giles proposed openstack/kolla-kubernetes: Fix typos in development environment documentation  https://review.openstack.org/42594622:03
*** jheroux has quit IRC22:03
inc0so not sure, they did shift focus to fuel-ccp, but I wonder what they downstreams think about them now22:03
inc0I expect lots of people feel screwed22:04
sdakeinc0 who is running forf uel ptl?22:04
inc0nobodyu22:05
sdakeseriously?22:05
inc0nobody submitted candidacy22:05
sdakesup v1k0d3n long time no see ;)22:05
sdakeis the deadline up?22:05
v1k0d3njust don't want new folks to think kolla is only concerned with world dominance ;)22:05
v1k0d3nmore community,  right? :)22:05
sdakev1k0d3n nah - we want everyone to work together22:05
sdakev1k0d3n when  I say we above, I really mean I22:05
sdakeI can't speak for others22:05
v1k0d3nsdake: i know.22:06
inc0well, I think we have good culture of this22:06
inc0anyway, Fuel PTL...that would be fun wouldn't it?22:06
sdakeinc0 kolla does, openstack in general needs improvement here22:06
sdakeinc0 if a project doen't have a ptl, it becomes a leaderless project and is ejected from the bigt tent22:07
*** unicell has joined #openstack-kolla22:13
inc0yeah, I wonder if that's going to happen22:13
inc0stupidest way to get kicked off22:13
inc0if you ask me22:13
*** sayantani01 has quit IRC22:21
*** sayantani01 has joined #openstack-kolla22:22
*** MasterOfBugs has joined #openstack-kolla22:23
*** unicell1 has joined #openstack-kolla22:23
*** sacharya has joined #openstack-kolla22:23
*** eaguilar has joined #openstack-kolla22:24
*** unicell has quit IRC22:26
*** mgiles has quit IRC22:29
kbaegisI'm former Mirantis myself :)22:30
kbaegisI'm shocked that no one submitted for ptl22:30
kbaegisThough ccp is the "new way forward" over there22:31
*** pramodrj07 has joined #openstack-kolla22:31
kbaegisNot the team I was on22:31
*** aric49 has quit IRC22:32
*** jrobinson is now known as jrobinson-afk22:36
bmace_sdake: fyi, getting very close now.  hit a recent helm issue because the version specified in the Docker container was wrong but once i fixed that in-container things were looking much better.22:39
*** crushil has quit IRC22:39
*** Syffs has quit IRC22:41
*** Syffs has joined #openstack-kolla22:41
*** esmiurium has joined #openstack-kolla22:43
*** sacharya has quit IRC22:46
*** msimonin has joined #openstack-kolla22:48
sdakebmace_ update from my end - i am able to run openstack keystone catalog22:49
sdakebmace_ there is a defect in kolla-kubernetes which makes openstack endpoint list not work22:49
sdakebmace_ there is a patch in teh queue for it22:49
sdakebmace_ otherwise, openstack seems to work22:50
bmace_sdake: excellent22:50
sdakeone problem I have is horizon is busted22:50
sdakeit works whe nyou hit the web page, however, after login it kicks out some odd error22:50
bmace_sdake: who needs a gui? ;)22:50
sdakebmace_ apparently people that write checks ;-)22:50
david-lylesdake: what is the error?22:50
sdakedavid-lyle moment i'll see if I still have the env up22:51
*** rmart04 has quit IRC22:52
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes: PS fixes issue discovered with keystone service package  https://review.openstack.org/42596422:52
sdakedavid-lyle https://drive.google.com/file/d/0B8q6xDPETSkHYkNHNi16TC0zSFE/view?usp=sharing22:55
sdakebmace_ ^^22:55
*** sayantani01 has quit IRC22:55
david-lyleseems like maybe the POLICY_PATH setting is wrong22:56
*** sayantani01 has joined #openstack-kolla22:56
sdakedavid-lyle which file is that set in?22:56
sdakedavid-lyle is that an env variable or something?22:56
david-lyleopenstack_dashboard/settings.py or openstack_dashboard/local/local_settings.py22:56
*** eaguilar has quit IRC22:56
david-lyleunless the path is really openstack_dashboard/openstack_dashboard22:57
david-lylebut that seems odd22:57
sdakesbezverk is the above patch to fix the problem found in keystone endpoint list?22:57
Pavosdake I was getting same error until I disabled selixun22:57
Pavoselinux22:57
sdakePavo pretty sure selinux is disabled22:57
bmace_sdake: i remember hearing recently there were some general horizon access issues on the trunk22:57
david-lyleif that path is correct, then it's saying nova_policy.json is not in the path22:58
david-lyle*directory22:58
david-lylenot path22:58
sdakepavo selinux is in permissive mode22:58
*** mgoddard has joined #openstack-kolla22:58
sdakedavid-lyle let me check the container contents - moment - and thanks for the ts help :)22:58
Pavoshould be good to go then22:58
sbezverksdake: not sure, I was deploying on my cluster keystone service and found some issues22:58
sdakesbezverk can you try keystone endpoint list on your personal cluster?22:59
sdakesbezverk i thought we moved to 3.y.z. containers22:59
sdakesbezverk b71ad3350d04        docker.io/kolla/centos-binary-horizon:2.0.2                     "kolla_start"            6 hours ago         Up 6 hours                                   k8s_main.7b85cb09_horizon-4073033154-5hfcp_kolla_b289a92d-e3e7-11e6-91de-003a7d69f5dc_95ba3a3a23:00
sbezverksdake: http://paste.openstack.org/show/596649/23:00
sdakesbezverk kfox1111 made a patch earlier for the endpoint list issue23:00
sdakesbezverk lathough I'd rather use yours23:01
sdakesbezverk would you mind comparing his to yours and ssee if its the same solution?23:01
sbezverksdake: I have not seen it23:01
sbezverkdo you have link?23:01
sdakesbezverk moment23:01
*** saneax-_-|AFK is now known as saneax23:02
sdakesbezverk https://review.openstack.org/#/c/425836/23:03
sdakeok - on to inspecint containers :)23:03
sdakedavid-lyle - i'm in the continaer, where shall I look?  find . -name openstack_dashboard -ls ?23:03
sdakedavid-lyle23:04
sdake()[root@horizon-4073033154-5hfcp openstack_dashboard]# pwd23:04
sdake /usr/share/openstack-dashboard/openstack_dashboard23:04
david-lylesdake: looking back at your paste23:05
david-lyleok what is in conf/ just below that23:05
david-lyle?23:05
*** srwilkers has quit IRC23:06
david-lyleand what is in /etc/openstack-dashboard if that is empty?23:06
sbezverksdake: please use kfox1111 it is more complete, he covers issue I have not hit23:06
kfox1111back.23:07
sdakesbezverk is his a superset of yours?23:07
sdakedavid-lyle ther is no conf dir23:07
*** lrensing has quit IRC23:07
sdakedavid-lyle this is in /etc/openstack-dashboard: ceilometer_policy.json  cinder_policy.json  glance_policy.json  heat_policy.json  keystone_policy.json  local_settings  neutron_policy.json  nova_policy.json23:08
sdakei think one problem is we are using 2.0.2 contianers when i thought we moved to 3.0.2 containers23:08
sdakekfox1111 can you confirm that kolla-kubernetes should e using 3.0.2 versoin of the containers if using master?23:09
sdake(the dev env is using 2.0.2, and iscsi_workflow config uses 2.0.2 for all version numbers)23:09
kfox1111trunk is still pointing to 2.0.2 by default.23:09
kfox1111as its the most tested.23:09
kfox1111we need to get the gate stable again, then play with enabling more 3.0.2 tests.23:10
sdakekfox1111 what was changed to 3.0.2?23:10
kfox1111and getting trunk working.23:10
sdakekfox1111 or did that patch not happen?23:10
sdakekfox1111 as far as I can tell, trunk is pretty stable for kolla-kubernretes - deployed htis morning and can do openstack catalog list atleast :)23:10
*** Penfold has quit IRC23:10
david-lyleok this, I think goes back to a patch I -1'd23:11
sbezverkkfox1111: I was deploying keystone service on my cluster and hit one issue, as a result submitted PS to fix it23:11
kfox1111sdake: 2 patches went in. 1 to move 3.0.1 stuff to 3.0.2. and a second to add full 3.0.2 support.23:11
* david-lyle looks23:11
kfox1111sdake: no, trunks broken.23:11
sbezverkbut it looks like you also worked in that direction23:11
kfox1111let me show you.23:11
kfox1111https://review.openstack.org/#/c/418550/23:11
kfox1111the t one is the trunk one.23:12
kfox1111http://logs.openstack.org/50/418550/60/check/gate-kolla-kubernetes-deploy-centos-binary-t-ceph-multi-nv/e47dc10/console.html23:12
sdakeright23:12
kfox1111volumes are broken.23:12
kfox1111though if that one issue is resolved, it should work I think.23:12
kfox1111I got everything else working..23:12
sdakehttp://logs.openstack.org/50/418550/60/check/gate-kolla-kubernetes-deploy-centos-binary-t-ceph-multi-nv/e47dc10/console.html#_2017-01-26_19_28_55_49658223:12
sdakegroan23:12
sdakekfox1111 is there a patch up for the volumes are broke nthing?23:12
kfox1111I haven't figured out why they are broken yet. :/23:13
kfox1111SamYaple thinks it may be the uid patch thingy.23:13
kfox1111:/23:13
sdakekfox1111 inc0 went on today and SamYaple as well about how kolla-ansible is cratered because of a change23:13
sdakeoh right23:13
kfox1111probably that same change. :/23:13
inc0still debugging23:13
*** absubram has quit IRC23:13
sdakei dont see how that would affect 2.0.2 images?23:13
kfox1111yeah.23:13
kfox1111it doesn't affect 2.0.2 images.23:13
kfox1111only trunk.23:13
sdaketrunk of which deliverable?23:14
kfox11112.0.2 works and 3.0.2 works.23:14
Pavowhat d you mean by cratered sdake ?23:14
sdakeoh got i t23:14
kfox1111only trunk.23:14
inc0doesn't look anything like my problem tho23:14
sdakepavo - a crater occurs when a meteor hits the surface at high impact- causing the meteor the exxplode23:14
inc0Pavo, I try to deploy trunk and my galera fails23:14
sdakecratering is the act of being the meteor :)23:14
Pavoah ok23:14
Pavoyeah I never have success with trunk, I only have success on pip package installing using kolla-ansible23:15
sdakekfox1111 cool - so fro mthe developer docs pov, the fact that trunk of the kolla images are broken is irrelevant23:15
sdakePavo right - trunk is not particularly stable because our gates are not particularlly stable23:15
sbezverkkfox1111: when you have a chance keystone thing.. here is what I was trying to get http://paste.openstack.org/show/596652/23:16
sdakehowever, they are getting better rapidly and with kolla-kubernetes gates, they are becoming superior23:16
sbezverkusing public_host variable set in values.yaml for service23:16
kfox1111sdake: right. out of the box, it should work as its pointing to 2.0.2, which is heavily gate tested.23:16
sdakeonce we add cross repo gating for kolla deliverable from kolla-kubernetes, our gating will kick ass and tkae names23:16
Pavofound a ansible playbook today that deploys 5 node k8s on openstack23:16
*** mgoddard has quit IRC23:17
sdakePavo ya there is a whole slew of those ;)23:17
kfox1111sbezverk: yeah, I've got a patch for that : https://review.openstack.org/#/c/425836/23:17
Pavoinclude kuberlet I think thats what its called23:17
kfox1111the problem is, 'openstack endpoint list' is an admin command.23:17
sdakedavid-lyle i know your busy - and i got distracted - any thoughts on how to proceed with horizon?23:17
kfox1111so it contacts the keystone public url you give it, looks up the endpoint for the admin endpoint, then contacts it.23:17
kfox1111if the cli command is outside of k8s, it sees 'keystone-admin' as the dns entry and has no idea how to get there.23:18
sdakepretty sure that was sambetts|afk 's original analysis :)23:18
kfox1111this ps makes it possible to put the keystone-admin service external, so that its contactable from outside the k8s cluster.23:18
kfox1111sdake: +1 on cross gating. should be pretty easy for kolla-kubernetes.23:19
kfox1111sdake: I just got periodic jobs into the gate. :)23:19
sdakekfox1111 nice - your our gating rockstar ;)23:19
kfox1111sdake: http://logs.openstack.org/periodic/ and search for kolla-kubernetes :)23:19
kfox1111we gotta get one patch landed, and then we will be caching the docker hub images for our own use. :)23:19
sbezverkkfox1111: I am ok but I also want to specify public_host for service23:19
kfox1111sbezverk: you can with that ps.23:20
kfox1111see: https://review.openstack.org/#/c/425836/2/tests/bin/ceph_workflow.sh line new side, 19723:21
Pavowell gotta get off here, gotta get on the plane23:21
Pavottyl23:21
david-lylesdake: try setting POLICY_PATH=/etc/openstack-dashboard in openstack_dashboard/local/local_settings.py23:21
sdakelater pavo23:21
sbezverkkfox1111: ok in this case I can drop public_host change from my ps and leave only change in vlaues.yaml23:21
sdakedavid-lyle moemnt let me give that a go23:21
Pavolaterz23:21
sbezverkkfox1111: you specify it for microservice, have you tested it for service package?23:22
sdakedavid-lyle #POLICY_FILES_PATH = '/etc/openstack-dashboard'23:22
sdakethats what is there now23:22
david-lyleremove the #23:22
sdakedavid-lyle have to keep in mind  2.0.2 = mitaka23:22
sbezverkkfox1111: for microservice overriding public worked, it did not work for service23:22
sdakeright23:22
kfox1111sbezverk: there shouldn't be any difference?23:23
sbezverkkfox1111: there was before the change23:23
kfox1111what was the problem you were seeing?23:23
sbezverksame type of issue we had with .Values.element_name not being acccessible in service23:23
kfox1111oh.23:23
kfox1111element name not using the global api?23:24
sbezverkthat is why I added search for public_host23:24
*** ipsecguy has quit IRC23:24
kfox1111but, why override elementname at all?23:24
sdakedavid-lyle ++ thanks dude23:24
david-lylesdake: all good?23:24
kfox1111yeah. search for public_host wsa part of the bug.23:24
sdakehey folks, are we copyign local_settings.py into the container properly for horizon?23:25
kfox1111though in my ps, I think I changed it to dns_name.23:25
kfox1111but I have the same fix.23:25
sdakedavid-lyle ya - pretty sure  we re not copying local_settings.py propery23:25
sbezverkkfox1111: my question was have you tested your PS with service?23:25
sdakedavid-lyle in other words, we rae using defaults23:25
sdakedavid-lyle defeect in kolla-kubernetes - not in horizon23:25
*** jrobinson-afk is now known as jrobinson23:25
sdakedavid-lyle thanks for helping sort it out :)23:25
kfox1111sbezverk: the gate tests all. ;)23:25
david-lylesdake: any time23:26
kfox1111check the service check if you doubt it. :)23:26
sbezverkkfox1111: I wish it was true23:26
sdakekfox1111 the gate doesnt test logging into horizon23:26
sbezverkbut it is not testing overriding public_host23:26
sdakekfox1111 i think testing all is a dream ;-)23:26
sdakekfox1111 i'd be happy with testing some :)23:26
kfox1111in all seriousness though, yes, the gate tests that: http://logs.openstack.org/36/425836/2/check/gate-kolla-kubernetes-deploy-centos-binary-2-helm-entrypoint-nv/a973add/console.html23:26
sdakekfox1111 it tests curling the horizon web address23:27
sdakekfox1111 but it doesn't actually log into horizon23:27
kfox1111sdake: true. true.23:27
sdakeif you actualy log into horizon, you get the following: https://drive.google.com/file/d/0B8q6xDPETSkHYkNHNi16TC0zSFE/view?usp=sharing23:27
kfox1111sdake: I'd like to change that at some point.23:28
kfox1111just havent gotten to it yet.23:28
sdakekfox1111 sounds good23:28
kfox1111I've done solinum tests before though.23:28
sdakekfox1111 in the mean time, can we sort out this horizon problem :)23:28
sdakekfox1111 i'm not quite sure how to copy local_settings.py into the container with k8s23:28
sdakekfox1111 i did it manually just now to get it going23:28
kfox1111sdake: interesting..... that use to work.23:28
kfox1111sdake: though it could have bitrotted.23:29
sdakekfox1111 pretty sure its the defaults rather then the copy23:29
kfox1111sdake: by configmap23:29
sdakekfox1111 one thing to keep in mind is 2.0.2 nad 3.0.2 hae different horizon structure23:29
sdakekfox1111 the config files are different iirc23:29
sdakedavid-lyle could confirm23:29
kfox1111ah. so could be broken for 3.0.2+23:29
sdakekfox1111 we have had this problem before23:29
sdakekfox1111 i am using 2.0.2 images, so its definately broken there23:29
sdakealhtough it may be working on 3.0.2 images23:30
kfox1111hmm...23:30
sdakei am not sure about the 3.0.2 thing23:30
sdakekfox1111 we had this problem with kolla-ansible with the shift of horizon from mitaka to newton23:30
kfox1111should be able to try 'helm upgrade kolla/horizon --set image_tag=3.0.2' and see23:30
sdakeok moment :)23:30
kfox1111to be honest, the last time I tried horizon config was with genconfig from kolla-ansible 3.x23:31
kfox1111so it might be a gregression in support with kolla-ansible 4.x23:31
inc0galera why u no working!?23:31
kfox1111sdake: http://paste.openstack.org/show/sfJBK6siBgTNgy92znOV/23:32
kfox1111^---- was generated a long time ago. I think that worked with 2.0.2.23:32
kfox1111see if yours is different?23:32
sdake[root@oscorehost1 kolla-kubernetes]# helm upgrade kolla/horizon --set image_tag=3.0.223:33
sdakeError: This command needs 2 arguments: release name, chart path23:33
sdake[root@oscorehost1 kolla-kubernetes]# helm upgrade kolla/horizon . --set image_tag=3.0.223:33
sdakeError: UPGRADE FAILED: error reading .venv/Kolla-Kube/include/python2.7: read /root/kolla-kubernetes/.venv/Kolla-Kube/include/python2.7: is a directory23:33
kfox1111get rid of the '.'23:33
sdakekfox1111 then it fails with the first error23:33
kfox1111oh. right....23:33
kfox1111what did you name your release?23:34
kfox1111(helm ls)23:34
sdakehow do I tell?23:34
sdakeok23:34
kfox1111and hope you named it something you can find. :/23:34
sdake[root@oscorehost1 kolla-kubernetes]# helm ls | grep horizon23:34
sdakehorizon-deployment                      1               Thu Jan 26 16:51:34 2017        DEPLOYED        horizon-deployment-0.4.0-123:34
sdakehorizon-svc                             1               Thu Jan 26 16:43:31 2017        DEPLOYED        horizon-svc-0.4.0-123:34
kfox1111great. so its named 'horizon-deployment'23:34
kfox1111so try: helm upgrade horizon-deployment kolla/horizon-deployment --set image_tag=3.0.223:35
*** ipsecguy has joined #openstack-kolla23:35
sdake[root@oscorehost1 kolla-kubernetes]# helm upgrade kolla/horizon-deployment --set image_tag=3.0.223:35
sdakeError: This command needs 2 arguments: release name, chart path23:35
kfox1111so, reviewers on: https://review.openstack.org/#/c/425836/ ?23:35
kfox1111sdake: still need the release name in there.23:36
sdakeoh i see23:36
sdakemoment23:36
sdakeeyes not doing well after eye exam23:36
sdakehaving trouble foucinng my eyeblals23:36
kfox1111no worries23:36
*** eaguilar has joined #openstack-kolla23:36
kfox1111(is everything but horizon working?)23:37
kfox1111(just wondering how close the dev env is)23:37
sdakekfox1111 i haven't tried booting a vm23:39
sdakekfox1111 havne't tried anything but endpoint list and catalog list23:39
sdakeso with that update no horizon is running via docker ps -a on either of my two nodes23:39
kfox1111sdake: ok. so up through keystone is known working.23:39
sdakewith the helm update23:39
kfox1111give it a few. its gotta pull an image23:40
kfox1111kubectl get pods --namespace=kolla23:40
kfox1111and look for a horizon pod.23:40
sdakeya in creating state23:40
kfox1111then: kubectl describe pod horizon-xxxx23:40
kfox1111then: kubectl describe pod horizon-xxxx --namespace=kolla23:40
kfox1111should tell you what its doing.23:40
sdake  4m            22s             10      {kubelet oscorehost2}                   Warning         FailedMount     MountVolume.SetUp failed for volume "kubernetes.io/configmap/47f5586a-e420-11e6-91de-003a7d69f5dc-logging-config" (spec.Name: "logging-config") pod "47f5586a-e420-11e6-91de-003a7d69f5dc" (UID: "47f5586a-e420-11e6-91de-003a7d69f5dc") with: configmaps "horizon-logging" not found23:41
sdakeimplementation looking pretty good from my pov23:42
kfox1111uh... did it forget about options previously set during an upgrade?23:42
sdakefew bugs here and there23:42
sdakekfox1111 no idea - still learning the ropes23:43
bmace_sdake: you have your own image repo you are pulling from i assume?  looks like the default is now 4.0.0-1 which will obviously not work when pulling from docker hub.. but at least everything up to the image pull is working well now.23:43
sdakekfox1111 hope to document atleast some of what i  learn :)23:43
sdakebmace_ pulling from dockerio23:43
bmace_sdake: which version tag?23:44
sdakebmace_ i am taking that script I pushed up and sambetts|afk 's worked on and starting it via kubeadm23:44
kfox1111sdake: I dont know either. but I expected values to come from the old version during upgrades. :/23:44
kfox1111asking the helm devs now.23:44
sdakebmace_ huh?23:45
sdakekfox1111 i see that - in slack :)23:45
sdakei've gotta jet to dinner in 15 mins23:45
sdakeso anyone have any qs or rquests for me lemme know now :)23:45
sdakemy kids need m0ar minerals23:45
sbezverkkfox1111: nope23:47
sbezverkyou need to specify all old values23:47
sbezverkfor upgrade, I learned that when was doing reconfguration with upgrade23:47
kfox1111sdake: for the upgrade, you have to specify all the values again. :/23:48
bmace_sdake: image tag version.  4.0 isn't available on dockerio is it?23:48
sdakecan someone spoon feed me the values :)23:48
sdakebmace_ the iscsi_workflow config uses 2.0.2 by default23:48
inc0SamYaple, still around?23:48
kfox1111helm upgrade ..... --set kube_logger=false23:49
*** eaguilar has quit IRC23:49
kfox1111I've got a much better way to do this in the future, but havent gotten a chance to merge it yet.23:50
sdakewhat are the elipses :)23:50
kfox1111once https://review.openstack.org/#/c/425836/ merges, I can finish up the work.23:50
kfox1111sdake: stuff from before.23:50
sdakesame command as before kfox1111 ?23:50
sdakeok23:50
kfox1111eyah.23:50
kfox1111yeah23:50
inc0kfox1111, maybe you can help me actually23:51
kfox1111inc0: maybe? whats up?23:51
inc0wsrep should have tables in mysql database right?23:51
sdakeok that deployed23:51
*** david-lyle has quit IRC23:51
kfox1111inc0: hmm. not sure. I know it has env values.23:51
kfox1111inc0: never looked if it had tables.23:52
inc0so I vaguely remember that is keeps some of its state in database23:52
sdakekfox1111 that upgrade command works23:53
sdakekfox1111 i'll let you know if horizon 3.0.2 images work23:53
kfox1111sdake: k. cool. thanks.23:53
kfox1111can you post your config.json for horizon?23:54
*** david-lyle has joined #openstack-kolla23:54
*** jrobinson is now known as jrobinson-afk23:54
sdakekfox1111 same error23:55
sdakekfox1111 pretty sure that config.json you pasted earlier was wrong - where di dit come from?23:55
kfox1111sdake: a 'kolla genconfig' from back in the 3.x days I think.23:56
kfox1111back when I had horizon working.23:56
sdakekfox1111 hmm ok23:56
kfox1111can't garantee that peraticular config worked, but shoudl be similar.23:56
kfox1111trying to figure out what changed.23:57
kfox1111we're for sure going to need a solinum test though.23:57
sdakewish i had more time to debug this - maybe have about 15 more mins23:58
sdakethe json file is different now in master configs23:58
sdakeit includes all of the policy json files23:58
sdakeand a /etc/openstack-dashboard/local_settings23:58
kfox1111yeah. mine only has:23:59
kfox1111kfox2@mantis:~/svn/kolla-kubernetes$ ls /etc/kolla/horizon/23:59
kfox1111config.json  horizon.conf  local_settings23:59
sdakelol @ hostname23:59
kfox1111:)23:59
sdakeok - so how do I get that local_settings into the config map?23:59
sdakeor where is that genreated?23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!